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Executive Summary 



This report presents the results of the formative evaluation of the Adult Learning, 
Literacy and Essential Skills Program (ALLESP). Data collection related to this 
evaluation took place between November 2008 and May 2009. 



Program Background 

Program Description 

On April 1, 2006, the Terms and Conditions of three main Programs (the National Literacy 
Program (NLP), the Office of Learning Technologies (OLT) and Learning Initiatives 
Program (LIP), formerly the Learning Initiatives Lund), were merged into one, ALLESP, 
in order to harmonize the objectives and outcomes from federal adult learning programs. 

The financial resources available to the Program total $155. 6M between 2006-07 and 
2010-11. As of April 1, 2007, the Office of Literacy and Essential Skills (OLES) under 
the Skills and Employment Branch is responsible for the implementation of ALLESP. 



Objectives and Activities 

The key objectives of the ALLESP are to promote lifelong learning by reducing non- 
financial barriers to adult learning and to facilitate the creation of opportunities for 
Canadians to acquire the learning, literacy and essential skills they need to participate 
fully in a knowledge-based economy and society. 

These objectives are accomplished by providing funding to eligible organizations to 
conduct the following four activities: 

• Knowledge generation, transfer and application; 

• Promotion of innovation: provision of targeted investments using innovative approaches; 

• Capacity-building: strengthening the capacity of adult learning and literacy sector and 
those involved in essential skills; and 

• Increasing awareness: adult learning, literacy and essential skills promotion. 

ALLESP targets adults already employed or preparing to enter the workforce, fa mi lies 
and communities with a particular emphasis on Aboriginal Canadians, immigrants, 
lower-skilled workers, and official language minority communities. ALLESP plays an 
indirect role rather than a direct role in improving Canadians’ skills, leveraging the 
activities of others through building on existing relationships and developing new 
partnerships with federal government departments, provinces, territories, business 
associations, labour and other stakeholders. 
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Evaluation Scope, Objectives and Methodology 

The evaluation covered the period from April 1, 2006 until March 31, 2008. The overall 
objective of the formative evaluation of ALLESP was to examine a number of key issues 
and questions pertaining to program design and delivery, performance monitoring and 
measurement and short-term outcomes. 

The formative evaluation was based on multiple lines of evidence, including the 
following methods: 

• A document review; 

• Key informant interviews with 10 internal representatives from among ALLESP manage- 
ment and program analysts, 10 external stakeholders comprised of six experts in the 
literacy and essential skills (LES) field who represent non-governmental organizations, 
and four other bodies representing provincial governments; 

• A review of administrative data; 

• A review of 15 project files; 

• A review of performance measurement systems and mechanisms; and 

• A survey of all ALLESP funding applicants (n=252), including funded applicants 
(n=145) and unfunded applicants (n= 107). 



Key Findings and Conclusions 

Program Design and Delivery 

Are ALLESP’s objectives and expected results clear? Is the logic model 
still valid? 



Validity and clarity of Program objectives and expected results, and consistency with 
current needs and priorities: The large majority of key informants viewed the overall 
objective of ALLESP to be valid, clear and appropriate, but perhaps too broad, far reaching 
and not as targeted as it could be. Meanwhile, some external informants were of the 
view that ALLESP’s focus was on LES as it relates to labour force participation, and 
this has come at the expense of overall social inclusion. This view appears to be 
shared by community representatives who, according to some internal informants, have 
resisted the shift from literacy as a single priority, to the new focus on linking LES and 
labour market participation. 

With respect to ALLESP’s consistency with current needs and priorities, interviews with 
key informants, as well as the review of documents revealed that ALLESP objectives are 
compatible with the needs of federal and departmental, OLES, and LES stakeholders. 
Some key informants raised concerns that perhaps the Program’s ability to respond to 
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regional priorities has been somewhat compromised, given the perceived lack of 
consultation with the Provinces and Territories (PTs), and the cancellation of the local 
and regional funding stream in 2006. 

Level of external awareness of ALLESP and its key activities: Overall, familiarity with 
the ALLESP and its key activities was low amongst external key informants, with OLES 
and the former National Literacy Secretariat (NLS) receiving the most recognition. When 
asked to identify the primary activities of ALLESP, the majority of funding applicants 
who responded to the survey were able to provide an answer. Among those who were 
able to, some believed ALLESP funds training, an activity not supported by the Program. 
It is also worth noting that 9% of funded respondents and 20% of unfunded respondents 
were unable to identify ALLESP’ s key activities. 

The extent to which ALLESP activities, outputs and outcomes seem appropriately 
identified, defined and logically linked; extent to which they can be achieved: While the 
logic model was acknowledged by internal informants to be a “work in progress”, in their 
view, there was a logical link between Program activities and outputs. External informants 
raised concerns regarding these linkages. In particular, while these informants believed 
that the activities themselves were appropriate, they questioned how they would lead to 
the development of the outputs and eventual outcomes. 

Program outcomes, as evidenced by most key informant interviews are appropriate and 
clear. Questions were raised as to whether the intermediate and longer-term outcomes 
could be achieved, given the limited size, resources and reach of the Program, as well as 
the challenges in tracking results. 

Extent to which partnership-based, multi-sectoral approach to improving LES, such as 
ALLESP, is the most effective way to achieve objectives: Developing sustainable 
partnerships with various partners and stakeholders within the LES community is seen as 
a key factor for successful Program delivery, especially given the Program’s small size. 
However, relationships between ALLESP and social partners (i.e., NGOs, not-for profit 
organizations) as well as economic partners (e.g. employers, sector councils, unions etc.) 
appear to have suffered since program integration, with external informants noting that 
there was a stronger and more equitable partnership orientation under NLS. Internal key 
informants reported that they were aware of these challenges and efforts by OLES to re- 
build these partnerships were ongoing. 

Is ALLESP being implemented as intended? 

Extent to which the initiative is implemented as intended: ALLESP implementation has 
undergone a number of changes in order to support revisions in Program direction, in 
particular with the introduction of essential skills as a key policy and program focus, and 
the subsequent emphasis on LES in the workplace. Despite these challenges, internal key 
informants reported that funded projects are aligned with the Program’s ultimate outcome 
of enabling Canadians to participate in the knowledge economy and society. The evaluation 
was unable to determine the extent to which funded projects are aligned with the 
Program’s activity areas due to reporting challenges identified in the CSGC. 
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Has the Program re-alignment under ALLESP led to effective Program 
integration, co-ordination and coherence? 

Extent to which the Program re-alignment under ALLESP has led to the effective 
integration of these previous Programs , as well as improved value, co-ordination and 
coherence: Overall, the findings from the key informant interviews, as well as the 
applicant survey, suggest that the re-alignment has yet to improve Program application 
and administration efficiency, with applicants reporting a decline in service when 
compared to the three previous programs. In addition, concerns were voiced by a large 
majority of external informants that the re-alignment had resulted in the loss of some of 
the most important aspects of the three previous programs, including NLP’s partnership 
approach with provinces and territories, OLT’s community networks, as well as research 
and innovation (e.g., action research). 

Clarity of new Program mandate, effectiveness of Program co-ordination, more 
effective programming, level of Program coherence: Evidence reveals a great deal of 
resistance at the community level towards shifting from literacy as a singular priority, to 
the additional focus on LES and labour market participation. As a result, there is the 
perception within the LES community that literacy is not an important issue for the 
Federal government and that it has lost its profile within the Department. 

In terms of the effectiveness of program co-ordination and programming, a few internal 
key informants recognized that there were ongoing organizational and cultural issues 
related to the integration of LES that may have impacted program co-ordination. Efforts 
are ongoing to integrate these areas under one organizational framework and address 
issues with the Program. 

Who are the recipients of ALLESP funding? 

Recipients of ALLESP funding: During the two year period covered by the evaluation, 
792 proposals were submitted. Of these, just under half (327 or 46%) of the proposals 
were rejected 1 and 1 16 or 15% were withdrawn. Quebec had the highest number of projects 
funded overall, followed by Ontario. In terms of the total amount of funding received, 
this was reversed with Ontario receiving the most project funding and Quebec second. 
Despite efforts to broaden the scope of organizations to include other eligible recipient 
groups including employers, academia, unions, etc., a large portion of ALLESP funding 
continues to be allocated to the not-for-profit sector. 

Target Groups of ALLESP -funded Projects: With approximately 69% of the information 
related to “Target Audience or Group” missing or blank, it is not possible to determine 
the target groups of ALLESP-funded projects based on the Common System for Grants 
and Contributions (CSGC). In addition, for those projects where information was available, 
the majority of target groups are referred to in general terms (classified as “adult learners”, 
or “other”), making it difficult to discern the specific groups targeted by these funded 



1 Rejected projects are classified in the CSGC as “Assessment - Rejected”;”Project Set-up - Rejected” or 
"Recommendation and Approval - Rejected” 
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projects. Survey findings indicate that projects tend to target multiple types of target 
groups, with the most common including low income households, the unemployed, families 
and Canadians in rural and remote areas. 

Key organizations, groups and initiatives that are missing or underrepresented 
i.e. not being funded and the impact on improvement of adult learning, literacy and 
development of essential skills: There is the belief amongst internal key informants that 
no key organizations, groups or initiatives are missing or underrepresented i.e., not being 
funded under ALLESP. This is contrast to evidence noted previously that the Program 
continues to overwhelmingly support the non-profit sector. In addition, a few external 
key informants indicated that smaller provinces and organizations may experience 
difficulties in obtaining funding, an issue that may be compounded by the cancellation of 
the local and regional funding stream in 2006. 

Are there adequate management and administrative systems in place for 
effective and efficient Program delivery? Could delivery be improved? 

Extent to which the current approach to determining priorities and distributing 
funding is the most efficient and effective way for ALLESP to achieve its objectives and 
consistency with the needs of the sector: Determining priorities as described by a few 
internal key informants is a multi-faceted process involving both research and consultation 
with external stakeholders. This process may have suffered due to the Program shifting 
‘too far’ away from a shared decision-making process with the Provinces and Territories. 
As previously noted, efforts are ongoing to re-build these partnerships. 

Extent to which eligibility criteria and application process for funding is clear and well 
understood: Program eligibility criteria and guidelines for the application process, as 
outlined in the various documents available to funding applicants, are generally clear and 
well understood as evidenced by the documentation review, key informant interviews, 
and the completeness of project files. In addition, 89% of funded applicants who responded 
to the survey found that ALLESP eligibility criteria and guidelines were clear and well 
understood. This is compared to 52% of unfunded respondents. 

Satisfaction with the Call for Proposal and Administration Process: Lorty-nine percent 
of survey respondents were satisfied or very satisfied with the ALLESP application and 
administration process overall, including 71% of funded respondents, and 28% of 
unfunded respondents. However, this level of satisfaction amongst funded respondents 
drops to 63% if they reported previous experience with the three previous programs. 
When asked to rate specific aspects of the ALLESP application and administration 
process, the survey found that respondents were less than satisfied with several areas. 
These include: the perceived lack of transparency surrounding the proposal review 
process (47% of funded respondents and 19% of unfunded respondents were satisfied or 
very satisfied); timeliness of funding decisions (43% of funded respondents and 17% of 
unfunded respondents were satisfied or very satisfied); and explanations provided for not 
receiving funding (only 18% of unfunded respondents were satisfied or very satisfied). 



Formative Evaluation of the Adult Learning, Literacy and Essential Skills Program 




Satisfaction with ALLESP Reporting Requirements: According to most key informant 
interviews, the need for Program accountability requirements are recognized within the 
LES community, and according to survey results, funded applicants are by and large 
satisfied with the ALLESP project reporting requirements. 

Adequacy of management and administrative systems in place, and extent to which 
the Program is delivered efficiently and effectively: Evidence collected through the 
document review, those informants currently involved with OLES, and the survey indicates 
that overall, most of the systems are in place to adequately manage and administer 
ALLESP. Several areas for improvement were suggested, including the need to: better 
communicate with applicants on application process changes and delays; promote proposal 
innovation and creativity by further encouraging funding applicants to view their 
proposals within the broader framework of LES; and improve horizontal coordination, 
information sharing and CSGC data tracking and extraction. 



Performance Monitoring and Measurement 

Is an adequate performance measurement mechanism and system in 
place? How could performance measurement be improved? 

Adequacy of performance measurement mechanism and systems: While a performance 
measurement strategy and indicators were identified for the Program at inception, efforts 
have been ongoing to further revise, validate and update the performance measurement 
and evaluation frameworks for OLES (which includes the component for ALLESP), as 
well as its reporting regime to more effectively capture data that can measure success of 
funded projects. 

To this end, OLES has developed two performance measurement templates for funded 
organizations. The first, designed for organizations who receive core funding, is the 
Performance Report and Environmental Scan Template (APR), which also includes a 
template for conducting an environmental scan. The second template, the Project Results 
Report (PRR), is designed for those organizations receiving project funding. 

Both templates capture a significant amount of performance information. As such, OLES 
will need to ensure that smaller organizations with limited resources are able to collect 
the information. In addition, given the difficulties in fully tracking project administrative 
data in the CSGC identified below, there is some question as to whether the internal 
capacity exists to compile, review and report on the data provided through these templates. 

Ways in which the performance measurement mechanism and systems could be improved: 
It was acknowledged by internal key informants that more work needs to be done to 
ensure effective accountability for results, including: developing meaningful targets for 
results; improved monitoring and reporting i.e. follow up at the end of funded projects to 
support outcome monitoring; and an active business plan. As previously noted, work is 
ongoing to develop a performance measurement framework that will take a number of the 
above issues into account. 
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Is the basic information necessary for the summative evaluation being 
collected? 



Information on funding applicants and recipients: Tombstone data captured on funded 
applicants and recipients appears to be 100% complete in the CSGC. This information 
includes: Organization name and mandate; Organizational Type or Category (i.e. not-for- 
profit, public or private sector); Mailing Address (including city, province, postal code); 
and other contact details. 

Information on project activities and outputs : While four streams of activity (research 
and knowledge; capacity building; innovative tools and practices; and promotion and 
awareness) are identified for the program in the 2006 ALLESP logic model, the CSGC 
has difficulty tracking these activities. What information is available regarding funded 
activities suggests that the majority of ALLESP-funded projects are focused on 
“strengthening capacity.” In terms of outputs, the CSGC identifies 19 “Venture” codes 
that capture the outputs of funded projects. These range from best practices, to educational 
and resource material, to research studies, to volunteer development. While organizations 
are providing this information, a review of the CSGC reveals that a high proportion of 
data is not being inputted into the system. It is worth noting that improvements in data 
entry are evident from 2006-07 to 2007-08 i.e. less blanks and missing information. 

Information on project outcomes: While the Program’s new PRR template provides 
funded organizations the opportunity to report on a project’s direct impacts (and thus the 
ALLESP’ s immediate outcomes), as is the case with most partnership-based funding 
programs, the ALLESP does not collect information on the ultimate users (i.e. learners) 
of funded projects. This, coupled with the fact that intermediate and ultimate outcomes 
by their very nature are acted upon by more than one player, make it difficult for the 
Program to determine whether or not it has achieved it intermediate and longer-term 
objectives. This will be dependent on the data collected by funding recipients themselves. 



Preliminary Findings - outputs oriented 

What type of Funding Agreements have been entered into under ALLESP? 

Type of funding agreements: Despite commitments made after the NLS evaluation to 
move away from grants, ALLESP evaluation findings indicate the Program is still 
heavily grant-focused. Over 266 (92%) of projects funding over the 2006-07 and 2007-08 
fiscal years were grant-based 2 , with the remaining funding source being contribution- 
based, with 22 projects funded (8%). In terms of the types of activities, streams and 
issues covered by funded projects, the CSGC database is incomplete, or lacks the 
necessary details to draw definitive conclusions. What information is available suggests 
that during the evaluation period, projects tended to focus more on literacy, rather than 
essential skills, and that there is a need to place more emphasis on LES as it related to 
labour market participation. 



2 The breakdown is as follows: 145 (94 %) for 2006-07; 121 (90%) for 2007/08. 
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Extent to which the development of literacy & essential skills research, tools and 
supports through ALLESP funding as well as the Program’s support of provincial and 
territorial coalitions and national organizations, contributed to, or is expected to 
contribute to ALLESP’s immediate outcomes: This evaluation noted that it is too early 
in the Program lifecycle to answer the full extent to which results have been achieved. 
Nevertheless, a number of ongoing issues were identified that may affect the development 
and achievement of ALLESP results. These include passive dissemination of project 
outputs (i.e., through websites) as opposed to targeted outreach; the need for additional 
resources for research and development; the sustainability of projects after funding ends; 
and the need to increase LES awareness, particularly in the workplace. 

Recommendations 

The evaluation resulted in the following four recommendations: 

1. It is recommended that Program objectives and activities should be revised to more 
accurately reflect the direction and scope of the Program and to ensure outcomes are 
achievable and measurable. 

2. It is recommended that the Program continue its efforts to communicate, support and 
consult with stakeholders and funding applicants in order to ensure openness and 
transparency with regards to its priority setting and decision-making. 

3. It is recommended that OLES continue with its commitment to improve its data 
collection systems so that sufficient and complete information is collected for monitoring 
and reporting purposes. In conjunction, it is recommended that proposal and reporting 
forms be updated to better reflect the CSGC codes and categories, thereby easing the 
burden on Program officers. Finally, recent revisions to the Program’s reporting 
templates will require ongoing communication and support for funding recipients. 

4. It is recommended that the Program investigate various ways of supporting the 
transfer of the knowledge regarding funded projects and their results. 



x 
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Management Response/Action Plan 



Introduction 

A formative evaluation of the Adult Learning, Literacy and Essential Skills Program 
(ALLESP) was conducted between November, 2008 and May 2009 and covered the 
period from April 1, 2006 to March 31, 2008. The objective of the evaluation was to 
examine a number of key issues and questions pertaining to program design and delivery, 
performance monitoring and measurement, and short term outcomes. 

On April 1, 2006, the terms and conditions of the National Literacy Secretariat, the 
Office of Learning Technologies and the Learning Initiatives Program were integrated 
into one, ALLESP, in order to harmonize the objectives and outcomes of federal adult 
learning programs. As of 2007, the program has been administered by the Office of 
Literacy and Essential Skills (OLES). 

As part of its goal to become a centre of expertise in matters related to literacy and 
essential skills, OLES uses the ALLESP terms and conditions through a grants and 
contributions model that funds projects intended to build knowledge and expertise; 
develop, adapt and disseminate tools, supports and best practices; and, sustain partnerships 
with literacy and essential skills (LES) stakeholders (e.g. provinces and territories, 
employers, unions and literacy organizations). LES programming uses this knowledge, 
expertise, tools and partnerships to leverage and influence skills investments by partners 
and to integrate and embed LES into existing training and skills development efforts. 
Project funding decisions are made by the department following a general call for 
proposals (CLPs) process as well as through strategic partnerships with key partners to 
identify and fill LES gaps. 

The present terms and conditions for ALLESP will expire in March 2011. A full review 
of ALLESP objectives, activities and outcomes is underway and will include any 
necessary adjustments to ensure the program terms and conditions reflect what is needed 
going forward. 



Summary - Key Findings and Conclusions 

Overall, interviews and document reviews revealed that ALLESP objectives were 
compatible with the needs of federal and departmental, OLES and LES stakeholders. 
Interviews further revealed that the overall objective of ALLESP is valid, clear and 
appropriate, but perhaps too broad and not as targeted as it could be. 

At the time of program integration in 2006, concerns were raised as to whether ALLESP 
would be in a position to provide adequate program service and administrative efficiency 
to clients. The overall findings concluded that program integration has yet to fully 
accomplish set goals around improved value, coordination and coherence. In addition, 
stakeholder maintained negative perceptions regarding the need for program realignment 
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and the shift from literacy as a singular priority to the focus on LES as it links to labour- 
force participation. 

In evaluating performance monitoring and measurement, the development of two reporting 
templates for funded organizations were found to be a significant improvement over 
previous data collection templates and should allow for the collection of a significant 
amount of performance information. However, there are still questions as to whether the 
internal capacity exists to compile, review and report on the data provided by these 
templates. As well, the Common System for Grants and Contributions database was 
deemed as incomplete in reporting types of activities, streams and issues covered by 
funded projects. 



Recommendations and Responses 

1. It is recommended that Program objectives and activities should be revised to 
more accurately reflect the direction and scope of the Program and to ensure 
outcomes are achievable and measurable. 

OLES agrees with the recommendation as stated. 

• During the evaluation period, OLES had instituted initiatives to address the perceptions of 
stakeholder groups on the scope of ALLESP funding opportunities. ALLESP’s objectives 
were seen as overly ambitious and far-reaching for a relatively small funding program. 
Outreach measures were taken to reduce these types of expectations of ALLESP. 

• Various definitions to the ALLESP Terms and Conditions were changed in 2008 to 
provide greater clarity and flexibility to project applicants. Specifically, eligible activities 
and costs were expanded to include essential skills-related activities being considered for 
grants as well as capital acquisitions related to leasing or purchasing equipment and 
supplies being approved as eligible costs. 



Planned Activities 

• ALLESP objectives and activities will be reviewed through a re-drafting of the program’s 
Terms and Conditions set to expire on March 31, 201 1. 

• OLES has begun preliminary work on this process. Four key areas for review and 
revision to address a shift in scope and focus have been identified in this early stage: 
key objectives; expected outcomes; eligible activities; and, eligible costs. 

• Early consultations took place in Spring 2010 with a first draft of the Terms and 
Conditions to be completed in Summer 2010. 

• Four policy framework sessions are currently being held, the goals of which are to 
examine OLES’ future directions in keeping with department and government priorities 
and objectives. 
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2. It is recommended that the Program continue its efforts to communicate support 
and consult with stakeholders and funding applicants in order to ensure openness 
and transparency with regards to its priority setting and decision-making. 

OLES agrees with the recommendation as stated. 

• During the evaluation period, OLES revised the CFP assessment process to include 
greater input from Provinces and Territories (PTs). The changes included providing PT 
representatives with executive summaries and budgets of proposed projects within their 
jurisdictions as well as conducting conference calls with each PT to gamer feedback on 
each project. The calls are meant to provide for open and frank discussion of proposed 
projects given PTs knowledge of the regional situation. 

• Upon OLES’ creation in 2007, visits were made to inform PTs of its new strategic 
direction as well as garnering feedback in the planning towards the first CFP process 
that was to be launched. 



Actions Taken 

• A series of visits to ALLESP-funded project sites were undertaken in Fall, 2009. 
These visits included PT government partners and allowed OLES to get a better 
understanding of LES from the regional perspective. 

• During 2009-2010, OLES convened three full-day meetings with National Literacy 
Organizations and PT Literacy Coalitions. Two of the meetings were of a general 
information sharing and consultative variety and included discussions on the ALLESP 
intake strategy and recent project results, and provided attendees with the opportunity 
to comment on the CFP process. The third meeting focused specifically on new 
performance measurement reporting processes instituted by OLES. 



Planned Activities 

• A strategic outreach plan is being developed to engage key stakeholders in discussions 
focused on OLES priorities, and assessment and decision-making processes. Stakeholders 
were solicited for feedback on the draft plan. It is anticipated that as a result of this 
plan, proposals will be more tailored towards ALLESP objectives. In addition, it is 
hoped that by engaging stakeholders in the development of the plan, OLES will be 
better informed as to their needs. 

• The full implementation of the strategic outreach plan is scheduled for Fall 2010. 

An additional issue was raised in the evaluation report, namely: “Efforts need to continue 
to recapture the trust and partnerships, especially between the two levels of 
government, with more effective and open communication and outreach.” 

• Since September 2008, OLES has held three multilateral, ad hoc meetings with PTs 
with representatives from Ministries of Education and Labour actively participating. 
During recent multilateral meetings and bilateral discussions, PT representatives have 
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commented on how much they appreciate the openness shown by OLES officials and 
the increased transparency which contributes greatly to FPT collaboration. 

• Beginning in 2009, OLES has actively pursued initiatives with specific PTs to develop 
and implement project activities outside of the CFP process via a ‘three-pronged 
approach’ to funding. Efforts to attract solicited and unsolicited project proposals are 
meant to complement the CFP process. These efforts are a product of ongoing bilateral 
discussions where shared priorities have been identified. 

3. It is recommended that OLES continue with its commitment to improve its data 
collection systems so that sufficient and complete information is collected for 
monitoring and reporting purposes. In conjunction, it is recommended that 
proposal and reporting forms be updated to better reflect the CSGC codes and 
categories, thereby easing the burden on Program officers. Finally, recent revisions 
to the Program’s reporting templates will require ongoing communication and 
support for funding recipients. 

OLES agrees with the recommendation as stated. 



Actions Taken 

• OLES has also developed an ACCESS database, a major, technical undertaking, to ensure 
that projects are more closely and more effectively monitored as well as to report 
specifically on what is being funded. ACCESS was developed to complement the CSGC 
which could not provide the needed flexibility in reporting that ALLESP projects 
required. ACCESS allows project officers to retrieve data using any number of criteria 
while being able to produce instant, real-time reports. 

• ACCESS also allows key groups that were targeted through our funding agreements to 
be more easily identified. Coding for thirteen specific categories allow OLES to know 
what has been funded for which groups. The key target groups include: Aboriginal 
Canadians; Apprenticeship; Employers; Immigrants; Linguistic minorities; Low skilled 
workers; Persons with disabilities; Practitioners; Sector councils; Seniors; Unemployed; 
Women; Youth. 

• To ensure that outcomes are achievable and measurable, OLES developed and implemented 
a comprehensive Performance Measurement Framework (PMF). OLES is currently 
collecting baseline information on a number of performance indicators and also 
monitors progress made on indicators for which it already has benchmarks. 

• As outlined in the Evaluation Report, the PMF requires new, more streamlined reporting 
from organizations receiving funding. Since April 2009, core funded organizations are 
required to complete an Annual Performance Report (APR), the objective of which is 
to better understand the organization’s overall performance by critically analyzing how 
well it is managed, its achievements and future plans. Concurrently, all other funded 
projects are required to complete a Project Results Report (PRR) that summarizes the 
outcomes of the project, the activities and outputs that will be carried out to achieve 
them, issues potentially affecting the project’s success and its current progress. 
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• In terms of uniformity: 

o OLES’ proposals and reporting forms have been updated to better reflect the CSGC 
codes and categories. 

o OLES’ monitoring reporting forms are the standardized ones used throughout the 
branch. 

o The proposals and concept paper templates being used have been adapted to the 
CSGC format and use the same core data items, as well as the same main categories 
(e.g. objectives, activities, budget). 

• Ongoing communication occurs with partners. At the onset of each project, an initial 
meeting is organized to clarify, among other items, the reporting requirements and the 
appropriate templates to use. OLES is also maintaining open lines of communication 
throughout the projects including regularly informing P/T literacy coalitions and other 
key partners of all changes that occur in our processes and templates. 



Planned Activities 

• OLES will develop a catalogue of results for APR and PRR reports that will be used to 
report on progress in enhancing LES, gain a better understanding of organizations 
receiving OLES funding and help in developing future calls for project proposals. 
A catalogue will be developed to receive input from both report templates by Fall 2010. 

4. It is recommended that the Program investigate various ways of supporting the 
transfer of the knowledge regarding funded projects and their results. 

OLES agrees with the recommendation as stated. 



Actions Taken 

• OLES has taken a proactive approach in furthering its knowledge transfer activities. 
In January 2010, OLES launched its new, re-designed website which includes a searchable 
project database. Stakeholders will now be able to use the website to gain knowledge 
about projects funded by OLES and can apply this knowledge to their particular 
situation. Comprehensive project results will also be provided online as they become 
available. The website also provides updated information on LES issues, resources, links 
and a special feature outlining specific, innovative items that represents the best of LES 
across the country. 

• Beginning in 2009, OLES instituted a regular cycle of meetings with key stakeholders 
to disseminate information and consult on LES issues from across the country. These 
meetings occur at least twice per year. Regular meetings have been convened with 
National Literacy Organizations, PT Coalitions and PT government representatives. 
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Planned Activities 



• OLES understands the importance of supporting the transfer of knowledge of funded 
project results. The APR and PRR reporting requirements will eventually produce a 
library of results data that will provide OLES with the requisite knowledge to help 
improve our programming. 

• As results are received, OLES’ will provide comprehensive, publicly-accessible project 
results on its website. For APRs, results will be posted by Fall 2010. For PRR results, 
results will be posted by Spring 2011. 

• As part of the OLES 2009-2010 business planning exercise, one of the priorities 
identified was the development of dissemination events with the goal to encourage 
dissemination of expertise through core funded partners. Three expertise-building fora 
are planned: 1) for eastern Canada, led by the Ontario Literacy Coalition, in Fall 2010, 

2) for western Canada, led by ABC Life Literacy Canada, in Spring 2011 and 

3) for northern Canada, led by the Yukon Literacy Coalition, in Summer 2011. 
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1. Introduction and Context 



This section provides a description of the context for the formative evaluation and brief 
overview of the Adult Learning Literacy and Essential Skills Program (ALLESP). 



1.1 Program Background 

Beginning in 1987, the National Literacy Program (NLP) worked to support activities 
that improve literacy skills in Canada using a partnership-based, multi-sectoral approach. 
The NLP worked with provinces and territories, businesses and labour organizations, the 
volunteer sector, universities and post- secondary institutions, provincial and territorial 
institutions and international non-profit institutions. NLP grants and contributions 
funded adult literacy and learning materials and models, public awareness, information 
coordination and dissemination, access and outreach, and applied research. 

In 1996, the federal government established the Office of Learning Technologies (OLT) 
as a partner in building a culture of lifelong learning. Its purpose was to enable Canadians 
to develop new knowledge and improve skills through the use of e-leaming by reducing 
barriers to the access and use of learning technologies. The four key activities of the 
OLT were: to promote and support the development and evolution of Community Learning 
Networks (CLNs); to help develop policies and strategies to guide the evolution and 
application of learning technologies and widely distribute the results; and to facilitate the 
sharing of information on Canadian initiatives, key players and significant developments 
in the application of learning technologies. 

The Learning Initiatives Program (LIP), formerly the Learning Initiatives Lund, was a 
contribution program established in 1994 to encourage and support initiatives that 
contributed to the development of results-oriented, accessible, relevant and accountable 
adult learning system in Canada. Through the LIP, Human Resources and Skills 
Development Canada (HRSDC) collaborated with key partners in the development and 
implementation of a wide array of national and international initiatives in support of 
lifelong learning. 

The NLP, OLT and LIP all had their core interest in development, promotion and support 
of lifelong learning in Canada that worked to enable Canadians to participate fully in a 
democratic knowledge-based society, both socially and economically. This supported the 
HRSDC’ s mission, which at the time was to “ improve the standard of living and quality 
of life of all Canadians by promotion the development of a highly skilled and mobile 
labour force and an efficient and inclusive labour market”. 
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1.2 Adult Learning Literacy and Essential Skills Program 
1.2. 1 Program Description 

On April 1, 2006, the terms and conditions of three main Programs: NLP, OLT and LIP 
were merged into one, ALLESP, in order to harmonize the objectives and outcomes from 
federal adult learning programs. The strategic goals of re-aligning the three Programs into 
ALLESP were described as: (1) increasing the effectiveness of the adult learning, literacy 
and essential skills sector to address learning needs of Canadians 3 ; (2) strengthening the 
operating base for monitoring projects, measuring, evaluating and reporting on results; and 
(3) maintaining and enhancing the Government of Canada’s capacity to play a constructive, 
visible role in the areas of adult learning, literacy and essential skills. 

More specifically, this realignment was expected to provide a cohesive and more solid 
Program for stakeholders and to further develop relationships around shared outcomes. 
In addition, while building on ongoing program successes, the program realignment was 
expected to address shortcomings that were identified in past evaluations of the former 
three programs such as enhancing research and information dissemination; increasing the 
targeting of particular groups 4 ; improving program coordination, coherence and support 
to literacy and essential skills (LES) through the identification of strategic priorities; 
establishing better outcomes tracking; and increasing the measurability of results. 

The ALLESP was initially designed to complement Learning Branch programs that 
addressed financial barriers to learning, such as the Canada Student Loans program and 
the Canada Education Savings Grant. This changed in September 2006 when, as part of 
the federal review of grants and contributions programs, federal resources for LES were 
re-targeted to focus on areas deemed to be in greatest national interest and core federal 
responsibilities while achieving concrete and measurable results for Canadians. As part of 
the re-targeting, the administration of the program fell under the newly created Office of 
Literacy and Essential Skills (OLES) under the Skills and Employment Branch in April 2007. 
Thus, under its four streams of activity (research and knowledge; capacity-building; 
innovative tools and practices; and promotion and awareness), organizations that apply 
for that funding would have to clearly demonstrate how their project proposals have a 
focus on the learner. This re-targeting resulted in a significant reduction of project funds 
($17.7 million over the two fiscal years under review which represented the federal- 
provincial-territorial partnership funding stream that supported community-based literacy 
projects). In response, the local-regional stream ended after the 2007-08 Call for Proposals 
(CEP). The national call remained in place and focused on LES in three areas: Business 
and Labour Partnerships and Workplace Literacy; Literacy and Learning Research; and 



3 ESI Pan-Canadian Community Outreach provided funding to community-based organization to inform families 
about the saving incentives available to their children and to raise awareness of the importance of saving early for 
their children's post-secondary education. This involved partnering with ALLESP to provide contribution funds to 
not-for-profit community groups to reach out to their constituents and stakeholders. Essential Skills Initiative was a 
separate entity when ALLESP was developed. 

4 The target groups of Aboriginal Canadians, immigrants, and low-skilled workers are in line with government 
priorities identified in the 2006 Speech from the Throne. 
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Integrating Literacy. With subsequent calls more emphasis was increasingly being placed 
on LES in the workplace as opposed to family and community literacy representing an 
important change for many in the LES community. 

ALLESP is delivered primarily through a CFP process 5 as well as core funding to over 
22 literacy and essential skills organizations across the country (which was introduced in 
the Fall of 2007). OLES also solicits proposals and accepts a small number of unsolicited 
proposals (outside the formal CFP process) for projects that fill an identified gap or need 
from organizations on the basis of their capacity, networks, activities, and expertise. 
The Office works closely with these organizations to help them complete a concept paper 
template. These submissions are then discussed and carefully analyzed through the 
Internal Assessment Committee (IAC), which is the same committee that reviews CFPs. 
Once approved by OLES, the normal process is followed for recommending a project to 
the Minister. 

In addition to the above-noted CFP process, in 2007-08, the Ministere de l’Education, du 
Loisir et du Sport (MELS) and HRSDC signed a one-year “Protocole d’ entente relative a 
l’alphabetisation”, which provided Quebec with $4.1M in ALLESP funding ($2.8M in 
contributions and $1.3M in grants) to fund province-based organizations to conduct 
literacy and essential skills projects for adult learners. This Protocole was signed following 
the 2002-03 to 2006-07 Protocole 6 and was designed to allow the Federal government to 
contribute to various literacy initiatives carried out by school boards and recognized 

' 7 

independent community action literacy organizations in Quebec. This resulted in an 
envelop of funds dedicated towards funding proposals submitted through the MELS and 
subsequently a large number of funded proposals originating from Quebec. 

To address this unique funding relationship, the Initiative federal-provincial conjoint sur 
la matiere d’alphabetisation (IFPCA) was a jointly managed body between Quebec and 
Canada. The IFPCA’ s funding for projects came from OLES via the ALLESP, with the 
MELS managing the project assessment process. This process involved a joint MELS-OLES 
staffed evaluation committee whereby recommended projects were sent to the Minister of 
HRSDC for final approval. 



1.2.2 Program Objectives and Activities 

According to Program documentation, ALLESP is expected to facilitate the creation of 
opportunities for Canadians to acquire the learning, literacy and essential skills they need 



5 The current process for posting calls for proposals is through the HRSDC website and direct communications with 
national partners and provinces as well as coalitions who communicate directly with their partners and stakeholders. 

6 Protocole agreements with provinces and territories were previously the norm, whereby there were clear funding 
envelopes that were spent by region. The final 5-year Protocole agreement with provinces and territories came 
following the 2002-03 to 2006-07 agreement. However, an agreement was reached with Quebec to continue the 
Protocole arrangement for 2007-08. 

7 The Act respecting the Ministere du Conseil executive (M-30) dictates that any organization (e.g. municipal bodies, 
school bodies or public agencies) in Quebec whose operations are partially or fully funded by the province of 
Quebec must obtain approval before signing any agreement with the Government of Canada, its departments or 
agencies, or a federal public agency. 
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to participate fully in a knowledge-based economy and society and promote lifelong 
learning through the removal of non-financial barriers. In particular, the Program is 
intended to strengthen the ability of practitioners across a variety of settings to effectively 
manage, deliver and evaluate LES. Support will enable practitioners to develop assessment 
tools, curriculum, and teaching materials, bring together best practices and make them 
available for tailoring to particular needs (provinces, territories, rural, urban, north, Aboriginal, 
workplaces big and small, different sectors). The Program is also expected to support 
professional development to increase the overall numbers and capacity of practitioners in 
communities and workplaces. 

Eligible recipients for contribution funding include 8 : non-profit organisations; professional 
associations; all organizations of the voluntary sector; provincial and territorial governments 
and their organizing bodies; provincial and territorial institutions including Crown 
corporations; universities, colleges and other educational and training bodies; workplace 
organizations, including sector councils, unions and business associations; international 
non-profit organizations (e.g. OECD); and, municipalities. Voluntary sector organizations 
are the only eligible recipients for grant funding 9 . 

ALLESP funding is provided for the following categories of eligible activities: 

• Knowledge generation, transfer and application; 

• Promotion of innovation: Provision of targeted investments using innovative approaches; 

• Capacity-building: Strengthening the capacity of adult learning and literacy sector and 
those involved in essential skills; and 

• Increasing awareness: Adult learning, literacy and essential skills promotion. 

The expected outputs and immediate outcomes of ALLESP correspond to the above 
categories of eligible activities (see logic model in Appendix A). 

ALLESP project funding is targeted towards those organizations that develop and share 
expertise and tools, designed to assist specific target groups such as Aboriginal people, 
immigrants, low skilled learners, official language minority communities, the homeless 
and persons with disabilities who are facing learning and literacy barriers to their 
participation in a knowledge -based economy and society in an effort to raise their LES 
levels. Funding applicants, as part of their proposals, are asked to describe the target 
groups for the project (e.g. literacy practitioners, adult learners, women, aboriginal 
Canadians) and indicate how they will benefit. When possible, they are also asked to 
provide an estimate of how many people will benefit. 



8 Sources: ALLESP Terms and Conditions for Class Contributions Consolidated Revenue Funds; Human Resources 
and Social Development - Adult Learning, Literacy and Essential Skills Program Applicant’s Guide (2006-2007); 
Terms and Conditions for Adult Learning, Literacy and Essential Skills; and, HRSDC website. 

9 Sources: ALLESP Terms and Conditions for Class Grants Consolidated Revenue Funds; Human Resources and 
Social Development - Adult Learning, Literacy and Essential Skills Program Applicant’s Guide (2006-2007); 
Terms and Conditions for Adult Learning, Literacy and Essential Skills; and, HRSDC website. 
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1.2.3 Program Governance 

The ALLESP is administered by the Office of Literacy and Essential Skills (OLES), of 
the Skills and Employment Branch 10 , HRSDC. Established in 2007, OLES’ mission is to 
become an acknowledged centre of expertise on “what works” in increasing the literacy 
and essential skills of adult Canadians. Using this expertise, OLES will play an indirect 
role by influencing the policies and activities of others and leveraging funding to improve 
opportunities for adults to increase their LES. As such, the OLES focuses on understanding 
what works in terms of improving literacy and essential skills; developing practical tools 
and supports to improve the literacy and essential skills of adults; and providing employers, 
practitioners and trainers with these tools and supports. 11 



1.2.4 Program Funding 

Budget allocations at program inception for ALLESP amounted to $31M in 2006-07 and 
$32M in 2007-08. Of these amounts, $24M was allocated to Grant and Contribution 
funding for 2006-07 and $25. 8M for 2007-08. The original budget breakdown over the 
five years is outlined in Table 1 below. “ 



Table 1 

ALLESP Original Budget by Vote 



(In millions of $) 



Fiscal year 


2006-07 


2007-08 


2008-09 


2009-10 


2010-11 


HRSDC Vote 1 (operating and salary) 


7.0 


6.2 


4.6 


4.5 


4.5 


HRSDC Vote 5 (Grants & Contributions) 


24.0 


25.8 


29.0 


26.5 


23.5 


ALLESP Funding Total 


31.0 


32.0 


33.6 


31.0 


28.0 



Vote 1 funding covers resources required to administer grants and contributions funding 
under the Terms and Conditions for the Employment Benefits and Support Measures 
(EBSM) under the purview of OLES as well as those required for ALLESP funding. 



1.2.5 Grants and Contributions Funding 

Actual ALLESP grants and contributions budget for 2006-07 and 2007-08 after re- 
profiling and internal re-alignment amounted to $24. 9M and $25. 7M respectively. Actual 
grants and contributions spending amounted to $24. 8M and $23. 3M respectively for a 
total of $48.1. 13 Of this amount, the project database review conducted for the evaluation 
was able to identify only $35. 2M in ALLESP grants and contributions spending. As such, 



10 During 2007-2008, the Department integrated the Programs under the labour market and workplace skills Program 
activities within the Skills and Employment Branch in order to more effectively promote a skilled, adaptable and 
inclusive labour force, and an efficient labour market. 

11 http://www.hrsdc.gc.ca/eng/workplaceskills/oles/olesindex_en.shtml 

12 Sources: Evaluation Terms of Reference 

13 Chief Financial Officer Branch, HRSDC 
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analyses conducted in this evaluation of grants and contributions funding is limited to the 
projects included in the $35.2M. 14 

Based on information contained within the Common System for Grants and Contributions 
(CSGC), between April 1, 2006 and March 31, 2008, a total of 288 projects were funded 
through the ALLESP. The vast majority (266 or 92%) of ALLESP projects were funded 
through grants. Only 8% of all ALLESP funded projects were based on contributions. 



1 .2. 6 Distribution of Funding 

Of the 154 projects funded in 2006-07, 84 (55%) received $75K or more from ALLESP, 
with 70 (45%) receiving less than $75K. In 2007-08, this was reversed, with the majority 
(52%) or 70 of the 134 projects funded receiving less than $75K. 

As a result of the Protocole with the Province of Quebec, Quebec accounted for the vast 
majority of smaller projects. Of the 140 projects receiving less than $75K from ALLESP, 
96 (69%) originated from Quebec. In addition, Quebec experienced a significant drop in 
the number of projects that received more than $75K between the two fiscal years 
analyzed as part of this evaluation. In 2006-07, 16 Quebec projects received more than 
$75K from ALLESP, a figure that dropped to four in 2007-08. It is interesting to note that 
if Quebec were to be removed from the analysis, of the remaining 172 projects funded 
during the evaluation period, approximately 70% would have received more than $75K. 



Table 2 

Size of funding allocation by Province 




(2006-07) 


AB 


BC 


MB 


NB 


NL 


NWT 


NS 


NV 


ON 


PEI 


PQ 


SK 


YK 


Total 


Under $25,000 


0 


0 


0 


0 


0 


1 


1 


0 


0 


1 


22 


0 


0 


25 


$25,000 -$74,999 


1 


2 


1 


2 


6 


0 


0 


0 


8 


1 


21 


2 


1 


45 


$75,000 -$199,999 


4 


4 


4 


3 


4 


1 


5 


0 


10 


1 


5 


3 


2 


46 


$200,000 + 


3 


3 


1 


2 


0 


2 


0 


1 


11 


2 


11 


2 


0 


38 


Total 


8 


9 


6 


7 


10 


4 


6 


1 


29 


5 


59 


7 


3 


154 





(2007-08) 


AB 


BC 


MB 


NB 


NL 


NWT 


NS 


NV 


ON 


PEI 


PQ 


SK 


YK 


Total 


Under $25,000 


0 


0 


0 


0 


0 


1 


1 


0 


1 


0 


27 


0 


0 


30 


$25,000 -$74,999 


1 


3 


0 


1 


2 


0 


0 


0 


4 


0 


26 


3 


0 


40 


$75,000 -$199,999 


5 


3 


3 


1 


1 


1 


0 


0 


9 


0 


0 


0 


1 


24 


$200,000 + 


5 


3 


4 


1 


0 


1 


0 


2 


18 


1 


4 


1 


0 


40 


Total 


11 


9 


7 


3 


3 


3 


1 


2 


32 


1 


57 


4 


1 


134 



14 The differences between departmental financial accounts and ALLESP projects database financial data could not be 
accounted for nor explained in the course of the evaluation and are still under review. 
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1.3 Evaluation Objectives and Scope 

The evaluation covered the period from April 1, 2006 until March 31, 2008. The overall 
objective of the formative evaluation of ALLESP was to examine a number of key issues 
and questions pertaining to program design and delivery, performance monitoring and 
measurement and preliminary findings and short-term outcomes. These questions were 
developed by the Evaluation Directorate of HRSDC, in consultation with OLES, and are 
as follows 15 : 

Evaluation Issue: Program Design and Delivery 

Key Evaluation Questions: 

• Are ALLESP’ s objectives and expected results clear? Is the logic model still valid? 

• Is ALLESP being implemented as intended? 

• Has the Program re-alignment under ALLESP led to effective Program integration, co- 
ordination and coherence? 

• Who are the recipients of ALLESP funding? 

• Are adequate management and administrative systems in place for effective and 
efficient Program delivery? Could delivery be improved? 



Performance Monitoring and Measurement 

Key Evaluation Questions: 

• Is an adequate performance measurement mechanism and system in place? How could 
performance measurement be improved? 

• Is the basic information necessary for the summative evaluation being collected? 



Preliminary Findings - outputs oriented 

Key Evaluation Questions: 

• What type of Lunding Agreements have been entered into under ALLESP? Do they include: 
o Multi-year funding of LES P/T coalitions and national organizations? 
o Projects that build knowledge about Canadians’ LES needs? 
o Projects that develop practical tools and supports to respond to identified needs? 



15 A complete description of the evaluation questions, indicators and methodologies used for this evaluation can be 
found in the Evaluation Matrix (Appendix B ). 
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2. Approach and Methodology 



The formative evaluation of the ALLESP was conducted using multiple lines of evidence. 
That is, more than one method was used to address each of the evaluation questions, 
thereby ensuring that the findings are corroborated. This section provides an overview of 
the main lines of evidence used to conduct the formative evaluation followed by a 
description of evaluation limitations. 



2.1 Key Lines of Evidence 

2. 1. 1 Document Review 

A review of Program documents and reports was undertaken to develop a profile of the 
Program, inform development of the data collection instruments, and to gather 
information on the Program design, delivery, performance monitoring and measurement and 
preliminary findings. All available and relevant documents and reports identified by the 
Project Authority and Program representatives were reviewed. These included a range of 
government documents such as departmental Reports on Plans and Priorities and Performance 
Reports, Speeches from the Throne, as well as those specific to the Program such as 
Terms and Conditions, and those related to program operations including guides for funding 
applicants, performance measurement framework, previous program evaluations, etc. 



2. 1.2 Key Informant Interviews 

A total of 20 key informant interviews were conducted between January 15, 2009 and 
February 13, 2009. Ten interviews were conducted with various representatives from 
among ALLESP management and Program analysts (herein noted as “internal'’ key 
informants) and ten were conducted with external stakeholders comprised of six experts 
in the LES field who represent non-governmental organizations and four other bodies 
representing provincial governments (herein noted as “external” key informants). 

Introduction letters were sent to all key informants prior to the interviews. The majority 
of interviews were conducted by telephone (with some in-person) in the official language 
preferred by the interviewee. Findings from each key informant interview were captured 
through detailed note-taking performed simultaneously during the interview. 

No information or findings has been attributed to any specific individual. Rather, all 
information provided was integrated with other interview results and presented in a 
summary format only using the following parameters: 
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“All or almost all” 


findings reflect the views and opinions of 90% or more of the key 
informants in the group 


“Large majority” 


findings reflect the views and opinions of at least 75% but less than 90% 
of key informants in the group 


“Majority or most” 


findings reflect the views and opinions of at least 50% but less than 75% 
of key informants in the group 


"Some" 


findings reflect the views and opinions of at least 25% but less than 50% 
of key informants in the group 


"A few" 


findings reflect the views and opinions of at least two respondents but 
less than 25% of key informants in the group. 




If, however, the size of the respondent population is too small, reporting 
of results has been limited to “the majority” and “a few” 



2. 1.3 Administrative Data Analysis 

The administrative database analysis was designed to review primarily quantitative 
evidence with basic frequency and cross tabular assessment. This component of the evaluation 
examined management information and administrative systems, namely the Common 
System for Grants and Contributions (CSGC), to determine their adequacy in supporting 
effective and efficient program delivery. 

The CSGC database maintains information on funding applicants to the Program, 
tracking information related to the proposal review process and decisions taken, limited 
information on unfunded or unsuccessful applicants, as well as information related to 
funding provided by HRSDC and from other sources. Officers are responsible for 
entering this information in the system based on their interpretation of proposals 
received. The information contained in this database is intended to be used to develop a 
profile of funding applicants, compliance monitoring, management and reporting. 

To the extent that data was available, the administrative database analysis provided data 
to support the review of Program activities, specifically: 

• Types of organizations funded and unfunded; 

• Types of projects funded; 

• Target audiences of funded projects; 

• Outputs from funded projects; 

• Timelines related to proposal review and decision-making process; 

• Distribution of funded projects by Region; 

• Applicant and partner contribution versus Program contribution, including in-kind 
contribution; 

• Key application process data; and 

• Project status. 
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Information contained in the CSGC databases was also assessed to determine the extent 
to which it is sufficient to support a subsequent summative evaluation. 



2. 1.4 Performance Measurement Systems and 
Mechanisms Review 

This component of the evaluation included an examination of the performance 
measurement systems and mechanisms (e.g. performance indicators, data collections 
procedures, data integrity and validity, reporting and use of results). The bulk of this 
review was undertaken as part of the documentation review and through the key 
informant interviews. 

Part of the current review included a more detailed assessment of ALLESP’s current 
performance measurement strategy as outlined in the Integrated Results-Based Management 
and Accountability Framework and Risk-Based Audit Framework for AFFESP 
(February 21, 2006) which outlined key performance indicators for monitoring; the approach 
used by the Program to support ongoing data collection and reporting was also assessed. 

It is important to note that the efforts have been ongoing during the evaluation period to 
further revise, validate and update the performance measurement and evaluation frameworks 
for OFES (which includes the component for AFFESP), as well as its reporting regime to 
more effectively capture data that can measure success of funded projects. This includes 
the development of the Core Funded Organizations 2008-2009 Performance Report and 
Environmental Scan template (APR), and the Results Reporting templates (for both grant 
and contribution agreements) (PRR). 

Work is ongoing to implement a new, comprehensive performance measurement and 
evaluation framework for OFES. 



2. 1.5 File Review 

In order to assess the Program’s performance management approach, 15 project files 
were randomly selected in order to review the information contained in relation to 
AFFESP and its key program areas and required file maintenance and reporting procedures. 
The quality of the following information was assessed, including: 

• Compliance with the application process and requirements; 

• Data contained in the file in contrast with the systems (e.g., appropriateness and accuracy 
of data input and capture); and 

• Whether the project is in compliance with the monitoring, evaluation and reporting 
requirements. 
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The 15 project files were randomly selected based on a set of selection criteria 16 , and 
consisted of 11 grant projects and four contributions projects. These projects, ranging 
from $3,000 to $623,492 in HRSDC contribution, are broken down as follows across 
provinces and territories: 



Table 3 

Number of Files and HRSDC Contributions 


Province/Territory 


Grant 


HRSDC 

contribution 


Contribution 


HRSDC 

Contribution 


Alberta 


1 


$53,791 






British Columbia 


1 


$41 ,544 






New Brunswick 


1 


$78,000 






Nova Scotia 






2 


$6,000 

$80,748 


Northwest Territories 


1 


$85,292 


1 


$3,000 


Ontario 


4 


70,220 

72,093 

190.000 

210.000 


1 


623,492 


Quebec 


2 


12,111 

17,000 






Saskatchewan 


1 


56,974 






Total 


11 


$887,025 


4 


$713,240 



2.1.6 Survey 

Contact Database Overview 

The CSGC database was used to create a list of potential survey respondents by selecting 
all accepted, rejected or withdrawn funding applications submitted between April 1, 2006 
to March 31, 2008. This resulted in 803 applications. This list was then purged to arrive 
at a list of potential survey respondents: 

• Among successful applicants, organizations appearing more than once were flagged 
and their most recently funded project proposal was considered for purposes of the 
study. As well, 6 projects funded under the Education Savings Incentives Program were 
removed from the database. A total of 214 organizations were initially categorized as 
“successful applicants”. 

• For the unsuccessful applicants’ database, all organizations having submitted at least 
one successful proposal were removed, ensuring that the unsuccessful database was 
composed uniquely of organizations having never received funding under ALLESP. 



16 Selection criteria - project budget (based on project distribution as follows: 25% under $25,000; 30% 25 but under 
$75,000; 30 over $75,000 but under $200,000; 15 % over $200,000); project was closed; and a representative 
sample of grants and contributions. 
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As well, 22 organizations appearing more than once were flagged and their most recent 
project proposal was considered for purposes of the study. As well, 64 projects that 
were classified as “other” were removed 17 . A total of 224 organizations were initially 
categorized as “unsuccessful applicants”. 

This resulted in a total of 438 applicants, and the resulting database was then analyzed to 
assess the following: (1) completeness; (2) accuracy; and (3) usability. Given the small 
size of the survey population, the entire population of applicants was asked to participate 
in the survey. 



Questionnaire and Question Design 

An iterative process involving the Evaluation Working Group developed individually 
tailored versions of the survey questionnaires for both groups being surveyed as part of 
this evaluation (successful and unsuccessful applicants). Two versions of each questionnaire 
were prepared - one for administration for a telephone interview, and one slightly 
modified for online data collection. 

Once finalized, the English and French telephone versions of the instrument were pre- 
tested with 9 applicants on April 2, 2009 for the purposes of quality control and finalized. 



Data Collection Process 

The process used for this specific study involved a variety of steps all aimed at ensuring 
the study was conducted with the appropriate individuals and ultimately to maximize 
response rates. First, on March 19, 2009, an advance letter, signed by senior HRSDC 
evaluation and ALLESP representatives, was sent by mail to potential respondents, 
informing them that they would be contacted shortly for the purpose of the survey. 
The letters were addressed to the “primary contact” as identified in ALLESP databases. 

The actual survey data collection was conducted using a multi-modal approach. The first 
“live” contact with respondents was made by telephone and provided an opportunity to 
assess the completeness, accuracy and usability of the contact database. In the event of 
discrepancies in the database, the interviewer was able to update the current database 
with proper contact information. Respondents were offered the opportunity to complete 
the survey either on the phone or online. If it was not a convenient time, the respondent 
was asked if they would like to make an appointment. 

As per industry standards, each valid telephone number was attempted up to 8 times 
before being considered “unusable.” To reach organizations that were considered 
“unreachable” by phone (e.g. answering machine, nobody answered the phone, etc.), a 
batch of emails inviting funding applicants to participate in the study was sent out on 
April 22, 2009. All respondents who were emailed an invitation to participate in the 



17 These 64 projects were not formally abandoned, rejected or withdrawn from the ALLESP databases - given their 
uncertain status, they were excluded from the potential survey respondent database. 
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survey but had yet to complete the survey were sent one wave of follow up email 
reminders on April 29, 2009. 

Data was collected over a one month period between April 6 and May 6, 2009. 
Ultimately, 252 surveys were completed (203 telephone surveys and 49 email surveys), 
of which 145 were completed with successful applicants and 107 were completed with 
unsuccessful applicants. This represents a “response rate” of 58% across all applicants. 
The survey took respondents on average 22 minutes to complete. 



Table 4 

Survey Data Collection and Response Rate 




Overall 


Funded 


Unfunded 


TOTAL Survey Frame 


438 (100%) 


214(49%) 


224 (51%) 




# 


Col. % 


# 


Col. % 


# 


Col. % 


Completed interviews 


252 


58% 


145 


68% 


107 


48% 


Nobody eligible to conduct survey 


26 


6% 


6 


3% 


20 


9% 


Respondent not available / No answer 


145 


33% 


62 


29% 


83 


37% 


Refusal / Termination 


15 


3% 


1 


<1% 


14 


6% 


Column Totals 


438 


100% 


214 


100% 


224 


100% 


Note: All percentages in this table are unweighted 



Analysis of the contact outcomes, as shown in Table 4, reveals that, while the overall 
response rate was 58%, this rate was higher among funded respondents (68%) compared 
to unfunded respondents (48%). Furthermore, unfunded respondents were three times 
more likely (9% versus 3%) than funded respondents to not have anyone at the organization 
qualified to participate in the survey (i.e. the primary contact was no longer with the 
organization). Unfunded organizations were generally harder to reach and less willing to 
participate, as seen by the 6% refusal rate among this group compared to less than 1% 
among funded respondents. 



Data Monitoring and Representativeness Considerations 

A census-type study of this nature with an overall response rate of 58% provided the 
research team a fair degree of confidence that the survey results reflect the opinions of 
the broader population. Both funded and unfunded applicants are also well represented. 
This confidence is further reinforced by the fact that the general profile of the respondents 
is very similar to that of the overall population across a variety of profiling information 
that was available in the CSGC, including by region, type of funding received, organization 
type, etc. 

It is worth noting that, as demonstrated in Table 5 below, there is a slight misalignment 
between funded and unfunded applicants overall. Specifically, funded applicants are 
over-represented - they represent 58% of respondents compared to 49% in the population. 
To adjust for the fact that funded organizations are over-represented, the applicant survey 
results are weighted to replicate actual population distributions by funding status 
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according to ALLESP population information. This weighting process will only have an 
impact on survey results presented for all applicants. Unless otherwise indicated in a note 
below a data table or figure, all percentages presented in this report are weighted. 



Table 5 

Data Monitoring and Distribution of Respondents by Region 




Population 

Distribution 


Respondent 

Distribution 


Funded 


49% 


58% 


Atlantic Canada 


7% 


8% 


Quebec 


18% 


23% 


Ontario 


11% 


12% 


Prairies 


7% 


6% 


British Columbia/Territories 


6% 


8% 


Unfunded 


51% 


42% 


Atlantic Canada 


8% 


8% 


Quebec 


5% 


4% 


Ontario 


11% 


9% 


Prairies 


9% 


7% 


British Columbia/Territories 


18% 


14% 


Total 


100% 


100% 


Note: All percentages in this table are unweighted 



2.2 Methodological Limitations 

As with any evaluation, there are a number of limitations to this study. The reader is 
encouraged to take these into account when reviewing the findings of this summary 
report. The main limitations associated with this evaluation are as follows. 



2.2. 1 Missing Data 

Of note is the fact that CSGC data for key areas of project funding were missing, 
specifically - close to two thirds of the electronic data was missing from the following 
five categories: 



Activity Area 

Under this category, there are four activity areas that the ALLESP funds, based on the 
four logic model activities. These include: 

• Knowledge 

• Promotion 

• Strengthen Capacity 

• Innovative Approaches 
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Funding Streams 

Under this category, the four funding streams identified below are based on the key 
settings in which funded activities will take place. These include: 

• Communities 

• Fa mi lies 

1 8 

• Educational Institutions 

• Workplace 



Issue Area 

Under this category, there are 15 codes which capture the key issue area related to LES. 
These codes range from basic skills and literacy, to labour and business, rural and isolated. 



Venture Code 

There are 19 codes that capture the outputs of funded projects and which range from best 
practices, to educational and resource material, to research studies, to volunteer development. 
These are all critical data sources that provide key information regarding the type of 
project that has been funded including the area funded, the tools and outputs. 



Target Audience 

Within the eligible key setting(s) selected for a project, funding applicants as part of their 
proposals are asked to describe the target groups for the project and indicate how they 
will benefit. When possible, they are also asked to provide an estimate of how many 
people will benefit. There are 25 codes for target audiences, which range from adult 
learners, immigrants, colleges and universities, literacy experts and practitioners, and 
provincial and territorial governments. 

Without complete information on the above-noted codes, it is very difficult to complete a 
full analysis of the type of projects that are being funded (including the area funded, the 
tools and outputs). As such, CSGC data could not be used to fully evaluate the extent to 
which the Program is funding projects that are aligned with priorities (its own and those 
of government), meet the needs of the LES community, and will have the greatest impact 
on LES of individuals in Canada - particularly in relation to the Canadian workplace. 
In addition, the data that is available within these five categories is heavily weighted on 
the province of Quebec, in particular for the fiscal year 2007-2008 given the 1-year 
protocol agreement as discussed under Sub-section “1.2.1 Program Description” . As such, 



18 While “Educational Institutions” is included as part of the original codes provided by the Program, it is not 
identified as a code within the CSGC. The assumption is that this code is no longer in the CSGC due to the shift to 
Workplace, Family and Community. 
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this data does not provide a representative picture of national efforts. It should be noted 
that marked improvements were observed from 2006-07 to 2007-08 in terms of the 
completeness of data for activity, stream, venture, target and issue. This could be partly 
attributable the more rigorous funding application process, which gathers information 
from recipients in a more systematic way. 

From a survey administration perspective and in terms of analyzing the results from the 
survey, the administrative database information shortcomings had a few noteworthy impacts. 

• First, although the evaluation was able to conduct some comparative analysis 
(i.e. comparison of distributions) to gauge the extent to which the distribution of survey 
respondents reflects the distribution of the full population among those for whom this 
information was available in the database, it is impossible to determine whether the 
distribution of respondents without this information is as good. While other, more 
complete data fields can be used to establish how similar the distributions between the 
respondents and the population are, the ability to extend this across all variables is 
hampered. 

• The gaps in the administrative data limited the consulting team’s ability to conduct 
post-hoc analysis using these profiling variables. As such, some of the analysis was not 
possible because of the database gaps. 



2.2.2 Survey Data Interpretation Limitations 

Based on the comparative analysis presented in the previous section and an overall 
response rate of 58%, the results for funded and unfunded respondents are very 
representative of their respective populations. The most important considerations that 
might have an impact on the validity of the results are associated with the following: 

• The nature of the study itself - the survey exercise by and large involves consulting 
individuals on an application and, in the case of funded applicants, an administration 
process that may have occurred a number of years ago. This study, and the confidence 
in the accuracy of the respondent’s opinions, depends perhaps inordinately more on 
each respondent’s ability to remember their experience than a typical research study 
would. At the same time, it is understood that given that amount of time that has 
elapsed, the possibility exists for hindsight bias on the part of respondents. 

• Halo-effect of unsuccessful proposals - the manner in which the final contact list was 
created resulted in a specific selection based on historical funding with the ALLESP. 
Those who had received funding at one point, were asked to focus on that particular 
project for the purposes of the study. However, it is entirely possible that these opinions 
may have been influenced by applicants’ other proposal submissions, which may not all 
have been successful. One cannot ignore that nearly one-third of funded applicants that 
participated in the study also submitted unsuccessful proposals since April 2006. While 
their opinions remain valid and valuable, caution must be exercised in trusting that these 
opinions are exclusively related to the project selected for the study. 
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Similarly, one might hypothesize that applicants who were successful would have a 
more positive opinion of the Program compared to those who submitted unsuccessful 
proposals. 

Finally, organizations who did not apply for funding were not included as part of this 
survey. As such, it cannot be said with any certainty why some groups and organizations 
might be under-represented or are not applying for funding. 
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3. Key Evaluation Findings 



Key evaluation findings are presented under each relevant evaluation issue and question. 

3.1 Evaluation Issue: Program Design and Delivery 

Are ALLESP’s objectives and expected results clear? Is the logic model 
still valid? 



Program Objectives 

The stated objective of ALLESP, as outlined in a variety of Program documents that were 
reviewed is “to promote lifelong learning by reducing non-financial barriers to adult 
learning and to facilitate the creation of opportunities for Canadians to acquire the 
learning, literacy and essential skills they need to participate in a knowledge based 
economy and society”. 

Validity and clarity of Program objectives and expected results: Evidence collected 
through a large majority of the key informant interviews revealed that the objective of the 
Program is valid, clear and appropriate, but perhaps too high level. In general, it is viewed 
as being too broad, and perhaps not as targeted as it could be. Specifically, the Program’s 
objective is viewed as too far reaching given its size and budget. A few internal key 
informant interviews also revealed that the word ‘opportunities’ may lead to the mistaken 
belief amongst funding applicants that the ALLESP is undertaking program delivery, 
when in fact this statement is viewed as being “more about leveraging opportunity” 
to support the development of more effective literacy and essential skills (LES) programs 
and tools and ultimately the increased acquisition of LES. 

Conversely, the Program’s focus of tying LES to economic objectives and labour force 
participation is viewed as too narrow by some external key informants who see the 
barriers to LES acquisition as more systemic - based on the notion of social inclusion. 
These external key informants noted that it is important to view LES not only as a means 
of enhancing productivity and facilitating labour force participation but also as a means 
for individuals to participate in society and enhance well-being. External key informants 
further emphasized this point by noting that there are a number of societal and contextual 
factors (e.g. poverty, ethnicity, education, belonging to a designated group) as well as 
language and cultural challenges impacting LES acquisition that are not addressed 
through a strict focus on LES in relation to labour force participation. Furthermore, there 



19 Sources: Terms and Conditions for Adult Learning, Literacy and Essential Skills; Human Resources and Social 
Development - Adult Learning, Literacy and Essential Skills Program Applicant’s Guide (2006-2007); 2006-2007 
Call for Proposal documentation (Business/Labour Partnerships and Workplace Literacy, National Call for 
Proposals, and Local Regional Call for Proposal - this excludes the documentation reviewed on the 2006-2007 Call 
for Proposals: Essential Skills Initiative which was a separate entity when ALLESP was developed); ALLESP 
Terms and Conditions for Class Contributions Consolidated Revenue Funds; ALLESP Terms and Conditions for 
Class Grants Consolidated Revenue Funds; December 2006 Draft Performance Measurement and Evaluation 
Frameworks of the Adult Learning, Literacy and Essential Skills Program; as well as on the HRSDC website. 
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is uncertainty and complexity related to the integration between ‘literacy and essential 
skills’ and how literacy might be more closely tied to employment goals. 

In terms of the Program’s aim to “ promote lifelong learning by reducing non-financial 
barriers to adult learning”, a few internal key informants revealed that there is limited 
evidence that the Program is really reducing non-financial barriers. Rather it was noted 
that the objective or focus of the Program is about developing knowledge and building 
expertise as to what works in building LES for Canadians to participate in a knowledge- 
based economy and disseminating this knowledge so that it is replicated elsewhere. 
This objective or focus is linked to ALLESP’s strategic outcome to “ enhance Canadian 
productivity and participation through efficient and inclusive labour markets, competitive 
workplaces and access to learning”. Furthermore, a few internal key informants noted 
that there is less emphasis on ‘adult learning’ (which is addressed under the Learning 
Branch) and more emphasis on building the LES that Canadian adults need for work, 
learning and life. As a result, these internal key informants question the use of “adult 
learning” in defining the Program objectives and furthermore in each of its key activities. 

Consistency of ALLESP objectives with the current needs and priorities of the LES 
community: The review of Program documentation supplemented by internal key 
informant interviews revealed that, in general, Program objectives are compatible with 
varied needs of the LES community. At the same time, concerns were raised by a few 
external key informants that provincial and territorial input into ALLESP priority setting 
may have suffered as a result of program integration. 20 

Furthermore, the majority of external key informants and a few internal key informants 
noted a number of LES community needs and priorities which may not be adequately 
addressed or supported by ALLESP funding. It should be noted that these recommendations 
need to be viewed in light of the fact that, due to jurisdictional authority, the ALLESP 
does not fund direct interventions or training in LES. With this in mind, key informants 
suggested the following: 

• Flexible programs and services which respond to the varied needs of a variety of 
marginalized learners e.g. key sub-populations in provinces (i.e. those between the ages 
of 15 - 65 without grade 12), who are more likely to be ineligible for employability- 
based LES training opportunities. 

• Access to (or the dissemination of) LES targeted materials and resources to: ensure that 
knowledge, tools and materials are being used to their full extent, and do not overlap or 
duplicate other efforts; support LES acquisition; and, foster a culture of learning. Tools 
and services must be developed to encourage individuals to view learning as an ongoing 
process. This is seen as an important way to address the psychological, emotional and 
social barriers that individuals face when trying to improve their LES. 

• Additional research and understanding in the following areas: 



20 For further details on how the shift to ALLESP has led to changes in how partners are defined, consulted and 
engaged please refer to the evaluation indicator: “Extent to which a partnership-based, multi-sectoral approach to 
LES promotion, such as the ALLESP, is the most effective way to achieve objectives” . 
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o The contextual factors surrounding LES and the uniqueness of the Canadian context 
(e.g., isolated communities and the impact this has on literacy levels); 

o The processes by which people attain essential skills, thereby assisting in determining 
which types of initiatives the ALLESP ought to fund; 

o Tools and supports at the community-level to measure performance of ALLESP- 
funded research, programming and tools 21 ; 

o Strategies to support LES development in the workplace amongst those who are 
already employed; and 

o Potential areas of collaboration (coordination between the federal, provincial and 
territorial governments) to support LES acquisition. 

In light of the fact that the ALLESP does not support direct delivery, several external key 
informants suggested that alternative policy instruments (such as El) ought to be developed 
instead to support Canadians who are interested in upgrading their knowledge and skills - 
particularly those who have been displaced from the labour market. 

Some additional emerging challenges faced by the LES community were identified including: 

• the replacement of an aging or retiring workforce in community -based programming; 

• an increase in service demands on community-based programs as larger numbers of 
learners try to access these programs in today’s knowledge based economy; and 

• the evolving needs of the LES community, which require access to ongoing practitioner 
training, and up-to-date curriculum. 

Consistency of ALLESP objectives with current federal and community needs and 
priorities: Overall, the ALLESP was viewed as being consistent with federal and 
departmental priorities as evidenced by key informant interviews and the documentation 
review. In particular, key informants viewed the Program as being aligned with federal 
government priorities regarding the importance of addressing national skills and employment 
issues, supporting workforce entry, transitioning of employees from declining sectors and 
helping Canadians gain and improve upon the LES they need to participate fully in a 
knowledge-based economy, communities and fa mi lies. 

More specifically, the Program’s objectives, key activities, outputs and outcomes are 
consistent with departmental priorities in relation to LES as outlined in the Department’s 
2008-2009 Report on Plans and Priorities (RPP). The 2008-2009 RPP states “[sjupport 
[for] workplace training for Canadians by working with provinces, territories and the 
private sector to make training and skills development more widely available to Canadian 
workers and better aligned with the needs of the economy. HRSDC will continue to 
encourage skills development and training in Canadian workplaces and to invest in the 
development of tools and other support mechanisms to raise LES of Canadians.” As part 



21 The documentation review supported by key informant interviews revealed that a number of ongoing efforts have 
been undertaken by the Program to improve upon planning, performance information, measurement and reporting. 
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of its program activity architecture, one of HRSDC’s strategic outcomes is to “ enhance 
Canadian productivity through an efficient and inclusive labour markets/competitive 
workplaces and access to learning achieved' . 

HRSDC’s strategic outcome is further supported by a number of complementary 
ALLESP activities such as “Labour Market”, “Workplace Skills” (including LES), and 
“Learning” - . These complementary activities provide Canadians with the opportunities 
and choices to participate in an array of programs. Although these program activities 
have distinct roles and responsibilities, they have common elements that contribute to the 
Department’s support of lifelong learning and the development and use of skills to work 
in the current and future labour market. 

In light of the above-findings, evidence from some of the key informant interviews 
revealed that there appears to be resistance at the community level (with smaller 
organizations and larger literacy organizations who have been leaders in the ‘literacy 
movement’) towards shifting from literacy as a single priority, to the new focus on LES 
directly linked to labour market participation. This resistance, as noted by a few internal 
key informants, may be the result of: 1) retargeting of federal investments and LES 
resources to meet national priorities (i.e., labour market participation) and 2) the lack of 
resources dedicated to the ALLESP 23 . As a result, there is the perception amongst a few 
key informants that literacy is no longer a federal priority, but rather the priority is 
essential skills as it relates to labour force participation. 

Consistency of ALLESP objectives with those of OLE S: Evidence from the majority of 
internal key informant interviews noted that there is consistency between ALLESP and 
OLES (i.e. logic models including objectives) even though they may use different 
wording. A few internal key informants indicated a need to review the ALLESP logic 
model in light of the revised OLES logic model to ensure it is reflective of the direction 
of OLES and the federal government. As noted by a few internal key informants, the 
ALLESP logic model is a sub-component of the OLES logic model - specifically 
ALLESP falls under OLES’s key activities of “Grant and Contribution” programming. 
The OLES logic model also includes other OLES activities such as partnership, labour 
market and analysis and dissemination. 



22 As noted earlier, the Literacy and Essential Skills program was transferred to the Skills and Employment Branch 
during the year in recognition of the importance of such skills among adults striving to find and retain employment 
as well as to families and communities. While full reporting on results is included under the Learning program 
activity, the program was fully integrated into the Labour Market and Workplace Skills program activities during the 
2007-2008 year. Source: Human Resources and Social Development Canada, 2007-2008 Estimates, A Departmental 
Performance Report. 

23 A review of Program documentation indicated ongoing funding cuts of $17.7 million ($5.8 million in 2006-2007 
and $11.9 million in 2007-2008) which represented the federal-provincial-territorial partnership funding stream that 
supported community-based literacy projects. ALLESP funds were reallocated to activities of national interest. 
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Program Activities and Outputs 

ALLESP identifies activities along the following four streams' : 

1. Generation, transfer and application of knowledge; 

2. Provision of targeted investments using innovative approaches; 

3. Strengthening the capacity of the adult learning and literacy sectors and those involved 
in essential skills; and 

4. Adult learning, literacy and essential skills promotion. 

These activities are accomplished through: the administration of contribution projects for 
learning, literacy and essential skills; grants (and leveraged grants) projects for literacy; 
and support for the national coordination of learning, literacy and essential skills issues. 

The outputs of these activities include: 

• Research and knowledge; 

• Pilot and demonstration projects ; 

• Partnerships, networks, tools, materials and associated resources; and 

• Public awareness products. 

Level of external awareness of ALLESP and its key activities: The evaluation found 
that familiarity with the ALLESP and its key activities was relatively low amongst 
external key informants, with OLES becoming increasingly recognized as the federal 
“face” ‘on the ground’. Those with prior involvement with the previously merged 
programs (LIP, OLT and NLP), as well as NLS 25 , were particularly critical of the level of 
ALLESP awareness within the LES community. According to these external key 
informants, there was a wider awareness of the NLS, and as a consequence, the federal 
government’s role and support in the LES community. In comparison, since the creation 
of ALLESP there is the perception within the LES community, according to a few 
external key informants, that efforts are no longer made to reach the broader LES 
community (e.g., the Program no longer advertises Calls for Proposals in newspapers, nor 
are press releases issued, both of which were standard practice under the NLS). 
Therefore, according to these key informants, there is a perception that only groups ‘in 
the know can apply’. This is a concern given that one of the re-alignment’s strategic goals 



~ 4 Additional Sources:; Human Resources and Social Development - Adult Learning, Literacy and Essential Skills Program 
Applicant’s Guide (2006-2007); 2006-2007 Call for Proposal documentation (Business/Labour Partnerships and Workplace 
Literacy, National Call for Proposals, and Local Regional Call for Proposal - this excludes the documentation 
reviewed on the 2006-2007 Call for Proposals: Essential Skills Initiative which was a separate entity when ALLESP 
was developed); ALLESP Terms and Conditions for Class Contributions Consolidated Revenue Funds; ALLESP 
Terms and Conditions for Class Grants Consolidated Revenue Funds; December 2006 Draft Performance Measurement 
and Evaluation Frameworks of the Adult Learning, Literacy and Essential Skills Program; HRSDC website. 

25 The key focus of the NLS, as the secretariat, was the administration of the NLP. Through the NLP, the NLS worked 
to promote literacy as an essential component of a learning society and to make Canada's social, economic and 
political life more accessible to people with weak literacy skills. It did this both through the NLP and also through 
research, outreach and working in partnership. 
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is to maintain or enhance the Federal Government’s visibility in the areas of adult 
learning, literacy and essential skills. 

In an attempt to evaluate the extent to which funding applicants are aware of and familiar 
with ALLESP, all survey respondents were asked what they believe to be the primary 
activities funded by ALLESP. Respondents were not provided with a specific list of 
responses from which they could choose. Rather this was an entirely open-ended exercise. 

While most survey respondents were able to provide some sort of answer, it is worth 
noting that 9% of funded respondents and 20% of unfunded respondents were unable to 
provide an answer to the question. This proportion was higher in Atlantic Canada (20%) 
and in Western and Northern Canada (23%). Among those survey respondents who were 
able to provide an answer, over one-third believed ALLESP funds activities related to 
essential skills training specifically or essential skills in general and 3 in 10 believe ALLESP 
promotes literacy training or literacy in general. These results are noteworthy given the 
fact that ALLESP does not fund training, beyond a few projects associated with “training 
the trainer”. This may be a further indication of the lack of familiarity among applicants 
for what ALLESP actually funds. Regarding other responses provided, approximately 
20% believe ALLESP provides funding to conduct and develop literacy programs. 
In addition, fewer than 20% of survey respondents indicated that ALLESP provides 
employment-related skills, followed by the promotion and improvement of adult literacy 
and the production and distribution of LES materials and tools. Other than the proportion 
of respondents who were unable to provide a response, results between funded and 
unfunded respondents are quite similar. 

Extent to which activities and outputs seem appropriately identified , defined and 
logically linked: Overall, evidence collected through internal key informant interviews 
suggests that there is a logical link between the Program activities and the outputs 
outlined in the ALLESP logic model 26 . In addition, for those internal key informants 
familiar with subsequent versions and revisions of the ALLESP logic model, the original 
logic model was seen as superior when compared to subsequent versions, which were 
viewed as drifting too far from Program reach. However, the original ALLESP logic 
model was still acknowledged to be a “work in progress”, something to be expected 
given that the ALLESP was trying to bring together three groups (NLP, OLT and LES) 
with a shared focus on literacy, but with different mandates, audiences, etc. 

External key informant interviewed found the linkages between the activities and outputs 
to be less clear. In particular, while the activities are written at an appropriate high-level 
and flexible enough to include a variety of activities required at the community level, it is 
unclear how the activities lead to the development of the outputs and ultimately the 
expected outcomes. Lor example, a few external key informants noted that activities and 
outputs are based on “soft approaches” such as knowledge transfer, influencing, 
awareness raising, but due to jurisdictional authority, the ALLESP does not fund direct 
intervention, delivery or training in LES. As such, there is uncertainty as to whether or 
not the Program’s outputs would actually lead to the expected outcomes. 
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Developed as part of the Terms and Conditions in February 2006. Please see Appendix C for Program logic model. 
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Program Outcomes 

Immediate Outcomes 

ALLESP activities and outputs are expected to lead to the following immediate or short- 
term outcomes 27 : 

• Improved dissemination, transfer and application of knowledge and information; 

• Increased capacity of funding recipients, other stakeholders and end-users or target 
groups; and 

• Increased awareness of the benefits of and opportunities for adult learning, literacy and 
essential skills. 

Intermediate Outcomes 

The expected intermediate outcomes of ALLESP are: 

• programming, services and policies that respond to evolving needs; and 

• Enhanced opportunities for adult learning, literacy and essential skills. 

Long-Term Outcomes 

The re-alignment of adult learning, literacy and essential skills programming and services 
that provide enhanced opportunities with increased awareness of the benefits of and 
opportunities for such programming and services, is expected to result in a long-term 
outcome of increased participation by Canadians in adult learning, literacy and essential 
skills. This is fully in line with HRSDC’s strategic outcome that through access to learning 
opportunities, Canadians can participate more fully in a knowledge-based economy and 
society. The latter would be the ultimate expected outcome of ALLESP. 

Appropriateness and Clarity of ALLESP outcomes; extent to which they can be achieved: 
Most key informant interviews found Program outcomes appropriate and clear. Similar to 
the predecessor programs, ALLESP is designed to achieve its objectives by working with 
and influencing organizations, groups and individuals to work in new ways to reach 
Canadians in order to improve overall LES. In light of its indirect role, it was questioned 
by a few key informants whether the Program’s long-term outcome of increased participation 
in adult LES can be achieved. In fact, some key informants noted that it is impossible for 
the Program to demonstrate the achievement of its intermediate and long-term outcomes, 
given that the Program doesn’t touch individual Canadians and there are too many 
extraneous factors at play that impact on LES levels among Canadians. This is partially 



21 Additional Sources: ALLESP Terms and Conditions for Class Contributions Consolidated Revenue Funds; ALLESP 
Terms and Conditions for Class Grants Consolidated Revenue Funds; December 2006 Draft Performance Measurement 
and Evaluation Frameworks of the Adult Learning, Literacy and Essential Skills Program; HRSDC website. 
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due to difficulties in measuring participation and tracking or measuring Program results, 
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as well as difficulties in collecting data on target groups of the funded projects.” 

From the perspective of some of the external key informants, the long-term outcome of 
“ increased participation by Canadians in adult learning, literacy and essential skills” 
should focus on the number of Canadians who have or could have real opportunity to 
increase their skill levels, or who have increased their skill levels because of ALLESP 
funding. Alternatively, some external key informants argued that what ALLESP is 
ultimately trying to improve is public awareness regarding LES and the quality and reach 
of LES programming. From these outcome perspectives, a few external key informants 
noted that the Program is far from achieving its stated long-term outcomes due to the 
challenges and limitations (as previously noted) imposed by the limited size 5 resources 
and reach of the Program. 

Extent to which a partnership-based, multi-sectoral approach to improving LES, such 
as ALLESP, is the most effective way to achieve objectives: Most key informants 
(internal and largely external) spoke of the weakness in the area of partnership since the 
creation of ALLESP especially with regards to communication and information sharing, 
when compared to the NLS. In particular, the reduced use of partners in defining priority 
areas of funding (particularly in the CFP and review process leading to funding decisions) 
and the communications and dissemination of information as to how funding is allocated. 

Some external key informants noted that the shift to ALLESP has led to a definite change 
in how partners are defined, consulted and engaged. For example, the feeling of equality 
once felt by provinces in terms of the efforts to work together through the various 
decision-making processes (e.g. priority setting, calls for proposals, funding) seems to 
have been replaced with a ‘stakeholder’ label and thus impacts the cooperative relations 
between the two orders of government. There is a perceived ‘guardedness’ around 
ALLESP now in terms of the way in which consultations seem to be more directed and 
one way rather than being open. 

Most external key informants noted that the NLS had a stronger and more equitable 
partnership orientation, with positive working relationships and coordination between the 
provincial, territorial and federal government, as well as practitioners. As such, these 
partnerships were viewed by some of the external key informants, as very effective and 
transparent, allowing the NLS to satisfy the unique needs of each of the provinces and 
territories by taking into account regional and contextual factors (e.g. population size, 
demographics, target groups). 

Despite the above-noted, a few internal key informants viewed the partnership process 
under the NLS as not supporting the overriding federal government priorities. At the 
same time, these interviewees acknowledged that the re-targeting of federal resources for 
LES in 2006 to focus on areas deemed to be in greatest national interest and core federal 
responsibilities has created Program challenges in terms of getting lost partners ‘back on 
board’ and securing those connections that were once forged through the discussion of 
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For further details please refer to evaluation indicator: “Extent to which the performance indicators accurately reflect 
outputs and results”. 
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ideas under previous programming structures. Some internal key informants noted that 
within the Program, efforts are ongoing to improve on and re-build partnerships and 
communications. 

Is ALLESP being implemented as intended? 

Extent to which the initiative is implemented as intended: Internal key informants had 
mixed views with regards to the implementation of ALLESP. While a few informants 
feel that funded projects are aligned with ALLESP objectives, other internal key informants 
stated that in their view ALLESP is a grants and contribution program under OLES and 
thus is simply a funding mechanism rather than a program that affects broader change. 
As such. Program implementation ought to be viewed as more indirect, with ALLESP 
playing a facilitation and enabling role by supporting projects that build partnerships and 
provide LES opportunities through the funding of research and tool development. 

The implementation of ALLESP has been impacted by a number of factors, according to 
most internal key informants, including: the on-going nature of program implementation 
to support changes in program direction; a limited Program budget which has restricted 
funding opportunities; time delays in getting Program performance measurements in place, 
as well as difficulties measuring and attributing project success; and, strained relationships 
between the provinces and the federal government. 

Has the Program re-alignment under ALLESP led to effective Program 
integration, co-ordination and coherence? 

Extent to which the Program re-alignment under ALLESP has led to the effective 
integration of these previous Programs, as well as improved value, co-ordination and 
coherence: Overall, a large majority of external key informants and a few internal key 
informants were highly critical of the program re-alignment. According to these informants, 
it is unclear as to why a Program re-alignment was needed or how the content of the three 
previous programs was integrated. This has led to the belief that the creation of ALLESP 
has led to the loss of some of the best and most powerful elements of previous programming. 
This includes NLP’s partnership approach with provinces and territories, OLT’s community 
networks, as well as research and innovation (e.g., action research - such as being able to 
test something new, developing a new tool, conducting research on whether or not 
methodologies applied to particular clients, groups etc). 

Another issue raised by those interviewed is the perceived decline in the ‘field presence’ 
of ALLESP in the regions, as a result of program realignment. Lor example, a few 
external key informants noted that under previous Program structures there was a greater 
connection and communications in the regions. This connection has greatly diminished as 
a result of the integration of the three previous programs in particular, the cancellation of 
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the local and regional funding stream in 2006 29 and a perceived weakening of partnership 

on 

and overall collaboration. 

Furthermore, findings from key informant interviews, as well as the survey, suggest that 
there is little evidence that such a re-alignment has improved efficiency. This was noted 
by delays in CFPs and funding decisions, diminished communications and connectedness 
in the regions as well as a perceived guardedness in which consultations seem fixed 
rather than being open, when compared to the three previous programs. 

A review of the administrative data found that lag time between project application 
received date and project start date, exceeded 6 months in 2006/07; this average declined 
by 60 days in 2007/08. The delay appears to occur between the receipt of the application 
and when it is entered into the CSGC. This delay in the review process was also 
highlighted during the key informant interviews. At the time the interviews were conducted 
(January - February 2009), it was noted that organizations who applied for ALLESP 
funding in September 2008 were still waiting to hear if their proposal was successful. 

Once a project is approved, funds on average, are released relatively quickly (15 days 
on average). 

A few internal key informants recognized the issues noted above, and highlighted the fact 
that program re-alignment occurred only recently. As such, improvements were expected 
as the Program matured. In addition, some internal and external key informants noted that 
the re-alignment of the NLP, OLT and LIP programs into one program was necessary to 
reduce duplication of efforts. In addition, by merging these three Programs into one, it 
removed the perception, as highlighted in the previous program evaluations, that each on 
their own was too small to have a real impact on the LES of Canadians. 

Clarity of new Program mandate, effectiveness of Program co-ordination, more effective 
programming, level of Program coherence: According to a majority of external key 
informants, there is a perception within the community that the ALLESP is focused on 
essential skills and employment, and that the community must ‘fight’ to keep a broader 
focus on literacy as a separate yet integrated issue. As such, there is the view within the 
LES community, according to a few internal and external key informants, that literacy is 
not an important issue and that it has lost its profile within the Department. 

In terms of the effectiveness of program co-ordination and programming, a few internal 
key informants recognized that the Program has experienced ongoing issues with regards 
to the program integration process that may have impacted program co-ordination, 
including: organizational factors (e.g. the change to the name and structure, the addition 



29 In 2006, as part of the federal review of grants and contributions programs, federal resources for LES were re-targeted 
to focus on areas/programs deemed to be in greatest national interest and core federal responsibilities while 
achieving concrete and measurable results for Canadians i.e. learners. As a result of these changes, HRSDC 
reprioritized its resources to focus on its national ALLESP funding stream. In fact, the 2006-07 ALLESP local and 
regional call for proposal was cancelled after the local and regional stream of the ALLESP was eliminated. 
Accordingly, none of the proposals submitted under this call were funded. The national call remained in place. 

30 For further discussions regarding the perceived weakening of partnerships under the ALLESP, please refer to the 
evaluation indicator: “Extent to which a partnership-based, multi-sectoral approach to LES promotion, such as the 
ALLESP, is the most effective way to achieve objectives” . 
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of OLES later in the integration process, changing physical locations etc.) and cultural 
factors (bringing together three groups with a shared focus on literacy and learning, but 
with different mandates, audiences, etc., followed by the introduction and integration of 
essential skills with literacy as a key policy and program focus). Despite this, a few 
internal key informants noted that the programs are now becoming more integrated and 
while there appears to be separate cultures between essential skills and literacy, it is also 
clear from discussion with the majority of internal key informants that OLES is making a 
concerted effort to integrate these cultures under one organizational framework. 

Who are the recipients of ALLESP funding? 

Applicants and recipients of ALLESP funding: Evidence collected from the administrative 
data analysis (Table 6) noted that of the 288 projects funded between April 1, 2006 and 
March 31, 2008, Quebec and Ontario received the largest proportion - a total of 
177 (62%) of all projects. Quebec accounted for the largest number of projects, with a 
total of 116, with Ontario accounting for a higher percentage (36% or $12. 5M) of the 
$35. 2M worth of HRSDC funding available during this timeframe. Quebec accounted for 
the second highest percentage with 21% ($7.2M). 

Only three other provinces (Alberta, British Columbia, Manitoba) and one territory 
(Nunavut) maintained roughly the same number of projects over the two-year period. Only 
Manitoba, Alberta and Nunavut experienced an increase in ALLESP project funding 
during this time, increasing by roughly $1.6M, $1.1M and $703,314 respectively. 

While the distribution of projects funded within various provinces remained relatively 
stable from year to year, Atlantic Canada experienced a significant drop in the number of 
projects funded, as well as ALLESP funding received, between 2006-07 and 2007-08. 
Combined, Newfoundland and Labrador, New Brunswick, Nova Scotia and Prince Edward 
Island received $2.8M for 28 projects in 2006-07. The total number of projects decreased 
to eight in 2007-08, with the ALLESP funding totalling just $962,948. 
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Table 6 

Distribution of Project Funding by Province by Fiscal Year 


Province 


(2006-07) 


(2007-08) 


#of 

Projects 


Total Project 
Funding 
Amount* 


ALLESP 

Funding 

Amount 


#of 

Projects 


Total Project 
Funding 
Amount* 


ALLESP 

Funding 

Amount 


Alberta 


8 


$1,518,956 


$1,012,837 


11 


$2,259,725 


$2,144,025 


British Columbia 


9 


$1,608,663 


$960,309 


9 


$1,453,960 


$1,099,251 


Manitoba 


6 


$889,023 


$681,363 


7 


$2,480,486 


$2,332,986 


New Brunswick 


7 


$973,367 


$816,386 


3 


$498,143 


$462,643 


Newfoundland and Labrador 


10 


$731,830 


$669,273 


3 


$210,738 


$199,738 


Northwest T erritories 


4 


$667,229 


$391,138 


3 


$400,724 


$288,695 


Nova Scotia 


6 


$661,625 


$596,265 


1 


$6,000 


$6,000 


Nunavut 


1 


$224,636 


$177,778 


2 


$953,212 


$881,092 


Ontario 


29 


$8,127,306 


$5,821,605 


32 


$7,783,681 


$6,689,883 


Prince Edward Island 


5 


$897,926 


$754,886 


1 


$294,567 


$294,567 


Quebec 


59 


$7,171,975 


$4,599,374 


57 


$3,103,869 


$2,639,611 


Saskatchewan 


7 


$1,003,110 


$921,925 


4 


$450,833 


$434,839 


Yukon Territory 


3 


$318,630 


$286,937 


1 


$75,191 


$75,191 


Total 


154 


$24,794,276 


$17,690,076 


134 


$19,971,129 


$17,548,521 


* Includes contributions from funding recipients and partners. 



Internal key informants noted that efforts are being made to broaden the scope of funded 
organizations to include other eligible recipient groups (employers, academia, unions, etc.). 
Despite these efforts, evidence from key external key informants and the administrative 
data analysis indicate that a large portion (213 or 74%) of ALLESP funding applicants 
and recipients are from the not-for-profit sector (specifically local community, charitable, 
voluntary organizations), followed by national NGOs and provincial NGOs (Table 7). 



Table 7 

Grants and Contribution Recipients 


Organization Category 


(2006-07) 


(2007-08) 


Total 


Aboriginal not-for-profit groups 




1 


1 


Associations of workers and/or of employers 


2 


5 


7 


Local community, charitable, voluntary org. 


117 


96 


213 


National NGOs 


14 


11 


25 


Provincial governments and agencies 




1 


1 


Provincial NGOs 


12 


11 


23 


Public community colleges and vocational schools 


1 


4 


5 


Public degree-granting colleges 


2 


1 


3 


Public degree-granting universities 




2 


2 


Public Sector (Incl. Univ./College/Vocational) 


5 


2 


7 


Unions 


1 




1 


Total 


154 


134 


288 
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Number and profile of rejected applicants: Of the 792 31 proposals submitted between 
April 1, 2006 and March 31, 2008, 46% (327) of proposals were rejected 32 and 15% (116) 
were withdrawn. Of the 327 proposals that were rejected, 85% (282) were within the 
not-for-profit sector, followed by 14% (46) in the public sector and only two projects 
within the private sector. Similarly, 87% (102) of withdrawn projects were within the not- 
for-profit sector, followed by 12% (14) in the public sector. No proposals were withdrawn 
from the private sector for either year. Only the Public Sector had more rejected proposals 
than approved proposals (of the 64 proposals submitted, only 17 were approved). Fewer 
rejected and withdrawn proposals were seen in 2007-08 compared to 2006-07. 

Target Groups of ALLESP-funded Projects ALLESP continues to experience 
difficulties in its ability to identify and track information on target groups as well as with 
consistently capturing this data within the CSGC. These findings are consistent with 
evaluation findings of previous programs. In fact, over 69% of the information on the 
“Target Audience or Group” was missing or blank in the CSGC. For a further discussion 
on the collection of target group information, please see “Is the basic information necessary 
for the summative evaluation being collected?” i.e. Profiles of the Target Groups. 



Of the 90 out of 288 projects for which information was available in the CSGC, 57% (51) 
targeted “adult learners” while 28% (25) were classified as “other” making it difficult to 
discern the specific target group of these funded projects. 



Table 8 

Profile of Target Audience or Group 


Target Audience or Group 


(2006-07) 


(2007-08) 34 


Total 


Adult Learners 


4 


47 


51 


Community Capacity Building 


1 




1 


Community Organization / Voluntary Sector 


1 


1 


2 


Educators / Trainers 


1 


6 


7 


Immigrants 




1 


1 


Learners / Students 




2 


2 


Other 


5 


20 


25 


Youth at Risk 




1 


1 


(blank) 


142 


56 


198 


Total 


154 


134 


288 



31 This total includes 288 successful projects, 327 rejected, 116 withdrawn and 61 projects in the CSGC classified in 
the CSGC under a variety of categories such as “Recommendation and Approval - In-process”; “Project Set-up - In- 
process”; “Agreement Set-up - Cancelled”; “Agreement - Cancelled” Assessment - Abandoned”; “Agreement Set-up - 
In-process”; and “Commit - In-process”. It is important to note however that these 61 projects were not formally 
abandoned, rejected or withdrawn from the ALLESP databases - given their uncertain status, they were excluded from 
our analysis of rejected/withdrawn applicants. 

32 Rejected projects are classified in the CSGC as “Assessment - Rejected”;”Project Set-up - Rejected” or 
“Recommendation and Approval - Rejected” 

33 Withdrawn projects in the CSGC are classified based on “Assessment - Withdrawn”; “Project Set-up - Withdrawn” 
or “Recommendation and Approval - Withdrawn”. 

34 Improvements have been noted for the 2007-08 in terms of more complete information i.e. less blanks over the 
2006-07 fiscal year 
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Survey findings revealed that individual projects tend to target multiple types of groups 
rather than focusing on a single group. Generally, low income households, the unemployed, 
families and Canadians in rural and remote areas are considered the most common target 
groups among both funded and unfunded applicants. Youth, immigrants and seniors 
represent the next tier of target groups with visible minorities (who are not new Canadians), 
aboriginals and official language minority (OLM) communities are less targeted. Results 
among both funded and unfunded applicants are quite consistent with the exception of 
unemployed Canadians, youth and visible minorities (who are not new Canadians) who 
were all more likely to be targeted by unfunded applicants. Given this latter finding there 
may be some cause for concern that a large number of proposals remain unfunded for 
projects which are targeted towards key groups who are unemployed and/or who need 
learning and essential skills development the most. 

Key organizations, groups and initiatives that are missing or underrepresented i.e. not 
being funded and the impact on improvement of adult learning, literacy and development 
of essential skills: Despite the views of internal key informants, who suggest that no key 
organizations, groups and initiatives are underrepresented under ALLESP, a review of 
the administrative data indicates that funding continues to be directed towards local 
community, charitable, voluntary organizations. This is despite efforts to target employers, 
academia, unions, etc. In addition, a few external key informants indicated that small regions 
(such as PEI) and smaller organizations (e.g. literacy-based) may experience difficulties in 
obtaining funding, an issue that may be exacerbated by the shift away from provincially 
designated funds under the ALLESP model, according to some external key informants. 
As previously noted, when compared to 2006-07, Ontario and Quebec combined experienced 
a 9% increase in the number of projects that received funding in 2007-08 (58% and 67% 
respectively). During the same period, the Atlantic provinces, which accounted for 17% of 
the funded projects in 2006-07, received only 6% in 2007-08 (see Table 6). 

Are adequate management and administrative systems in place for 
effective and efficient Program delivery? Could delivery be improved? 

Extent to which the current approach to determining priorities and distributing 
funding is the most efficient and effective way for ALLESP to achieve its objectives and 
consistency with the needs of the sector: Determining priorities, as described by a few 
internal key informants, is a multi-faceted process involving both research and consultation 
with external stakeholders. These consultations with external stakeholders, particularly the 
provinces and territories, appear to have suffered due to a shifting ‘too far’ away from a 
shared decision-making process. Despite this, most internal key informants discussed 
efforts the Program is making to continue to improve on and re-build partnerships and 
communications with provincial and territorial partners in the priority setting process to 
ensure the Program remains relevant to the LES community e.g. targeted calls for proposals. 
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Extent to which eligibility criteria and application process for funding is clear and well 

'JC 

understood: Program eligibility criteria and guidelines for the application process, as 
outlined in the various documents available to funding applicants, are generally clear and 
well understood as evidenced by the documentation review, key informant interviews, 
and the completeness of project files as part of the file review. Survey findings indicate 
that 62% of respondents overall were satisfied with the clarity of eligibility criteria 
(see Figure 2), including 84% of funded respondents. The level of satisfaction drops 
amongst unfunded applicants, with only 52% reporting being satisfied with program 
eligibility criteria. 

In an effort to ease the application process, supports and tools were introduced in 2007 as 
part of the CFP Application Guide and Kit. These include checklists intended to ensure 
key forms, documents, and mandatory information requirements are provided and to 
certify that the information contained within the project proposal is an accurate 
description of the organization, their plans and activities. Despite these supports and 
tools, only 46% of all funding respondents (including 58% of funded respondents and 
34% of unfunded respondents) are satisfied with the “ease of the funding application 
process” and 45% of all applicants (60% of funded respondents and 32% of unfunded 
respondents) are satisfied with the “assistance provided at the proposal development stage.” 

When asked to rate the assistance provide by ALLESP at the proposal development stage, 
some 17% responded don’t know/not applicable, suggesting that awareness and/or use of 
ALLESP assistance is not universal. This indicates that perhaps there is a lack of 
awareness of the available tools and supports amongst some funding applicants, or 
perhaps they may not be as effective and useful as is currently assumed. 

It is also worth noting that evidence collected through a few internal key informant 
interviews suggests that the application process may not do enough to communicate to 
and encourage funding applicants to propose projects within the broader framework of 
LES. As a result, some organizations, particularly the smaller ones, tend to continue to 
submit proposals that look to address literacy and/or essential skills more narrowly and 
perhaps less innovatively. It was suggested that more could be done through the CFP 
process (i.e. communications, outreach and promotion) to encourage innovation and creativity. 

Satisfaction with the Call for Proposal and Administration Process: Evidence collected 
through key informant interviews and the survey indicate a low level of satisfaction with 
the ALLESP application and administration process, especially amongst those organizations 
who did not receive funding. As demonstrated in Figure 1, only 28% of unfunded 
respondents were satisfied with the application and administration process overall, compared 
to 71% of funded respondents. This level of satisfaction drops to 63% amongst successful 
applicants who had previous experience with ALLESP’ s predecessor programs (such as 
the NLP, OLT and LIP). 



35 Guidelines are consistent with eligibility criteria and the guidelines for the application process for funding outlined in a 
variety of Program documents including: the RMAF/RBAF, the ALLESP Terms and Conditions for Class Contributions 
Consolidated Revenue Funds, and Terms and Conditions for Class Grants Consolidated Revenue Funds. 
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Aspects of the application process that received the lowest scores (at or below 31% 
satisfaction overall) are all related to the post-submission phase of the process. Satisfaction 
levels for the “transparency of the proposal review process” and the “timeliness of funding 
decision after the proposal submission” were both below 50% for funded respondents and 
below 20% for unfunded respondents. In addition, only 19% of unfunded respondents 
were satisfied with the “explanation provided for not receiving funding.” It should be 
noted that organizations who were denied funding in 2007-08 were less satisfied with the 
explanation received than those who were denied funding in 2006-07 (only 12% were 
satisfied in 2007-08 compared to 26% in 2006-07). This is counter-intuitive, given that as 
part of the post-mortem that took place after the first CFP in 2006-07, it was decided that 
all future applicants who were not approved for funding would receive a personal call 
from a Program representative detailing why they did not receive funding, and what 
improvements could be made for future calls, in addition to a formal rejection letter. 
Given this decline, it would seem that Program efforts to follow up directly with 
unfunded applicants did not lead to an increase in the level of satisfaction with the 
reasons for not receiving funding. 
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Figure 2 

Satisfaction with Specific Aspects of ALLESP Application and Administration Process 

Overall Applicants 



Timeliness of funding receipt after proposal approval* 
Availability of info regarding CFPs 
Assistance provided during project implementation* 
Clarity of eligibility criteria forprojectfunding 
Assistance provided atprojectdesign & agreementdevelopmentstage* 
Clarity of proposal review criteria 
Ease of funding application process 
Assistance provided atproposal developmentstage 
T ransparencyof proposal review process 
Timeliness of funding decision after proposal submission 
Explanation provided for not receiving funding** 




0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



□ Very satisfied ■ Satisfied □ Neutral ■ Dissatisfied ■ Very dissatisfied 



Source: C2 Please rate your level of satisfaction with regards to the following aspects of the ALLESP application 
and administration process. 

* Only asked to Funded Applicants 
** Only asked to Unfunded Applicants 



The slow turn-around times related to the processing of applications on the part of OLES 
was a significant issue raised by survey respondents. It is interesting to note that 
applicants who received funding under the previous three programs were particularly 
frustrated with the application and administration process, which indicates that one of the 
main reasons for the Program’s creation (based on the integration of the three programs) 
has yet to be accomplished. This low level of satisfaction with the ALLESP application 
and administration process is supported by evidence collected through key internal and 
external key informant interviews and the administrative data analysis, which revealed 
delays in the CLP process and an overall decline in service. 

Satisfaction with ALLESP Reporting Process: According to survey results, funded 
respondents are largely satisfied with the ALLESP project reporting requirements. Over 
three-quarters (78%), are either very satisfied or satisfied with these requirements while 
only 6% are dissatisfied. The remaining 12% expressed a neutral score on this matter. 
Among those who were dissatisfied, the most common complaint was that reporting 
requirements kept changing throughout the life cycle of the project, causing a number of 
resource and timing conflicts among funding recipients. 

It is worth noting that the ALLESP funds the majority of its projects through grants, 
which involve fewer accountability and reporting requirements. According to a few 
internal and external key informants, contribution funding requirements have increased 
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the administrative and reporting burden for organizations, particularly for smaller 
organizations who do not have the same organizational structure and resources as larger 
organizations (e.g., a university). OLES will have to bear this in mind as it moves away 
from grant funding towards contributions. 

In an effort to address the Blue Ribbon Panel on Grants and Contributions’ 36 recommend- 
ations regarding the need to make the delivery of grant and contribution programs more 
efficient and less burdensome, while ensuring greater accountability, a number of 
improvements have been made to the Program’s performance framework, planning and 
reporting regime since mid-2008. 37 These efforts are targeted towards improving 
accountability and performance reporting amongst funding recipients at the community 
level and consistency across regions while providing them with the necessary supports 
and tools to accurately measure performance of research, programming and tools that 
contribute to learner gains and outcomes. This includes the introduction of a Project 
Results Report (PRR) template (for both contribution and grant agreements) designed to 
streamline reporting while at the same time improving accountability. It also includes the 
Core Funded Organizations 2008-2009 Performance Report and Environmental Scan 
(APR) designed to assist core funded organizations in reporting overall performance. 

Survey results indicate that over half of funded respondents are either not aware of 
ALLESP’s previous performance measurement collection tools, or are not satisfied with 
the tools provided. With regards to the latter response, when asked what aspect of the 
performance measurement collection tools they were dissatisfied with, respondents reported 
that they were simply unaware of them. A key success factor for the above-noted 
improvements, as noted through survey findings, will be effective communication to 
funding applicants and recipients to increase awareness of these tools moving forward 
and to ensure the reporting process is simple and not overly onerous for them. 

Adequacy of management and administrative systems in place; Extent to which the 
Program is delivered efficiently and effectively: Evidence collected through the survey 
and a few internal key informant interviews indicates that the management and adminis- 
trative systems are adequate to meet most of the Program’s needs. Despite this, a number 
of weaknesses and potential areas for improvement were identified through the key 
informant interviews and survey. These include the need to: 

• Improve horizontal co-ordination, alignment, communication and information sharing 
across all Program areas to have, for example, a common understanding of OLES and 
how it sets out to achieve its outcomes while communicating a consistent message to 
the LES community. 



36 Through the Federal Accountability Act and its Action Plan, the Government of Canada is bringing forward specific 
measures to help strengthen accountability and increase transparency and oversight in government operations. The 
Action Plan aims to strike an appropriate balance between oversight and flexibility. The goal is to create a policy that 
helps ensure the achievement of desired outcomes, improves accountability, and encourages management efficiency. 
Accordingly, in June 2006, an independent blue ribbon panel was commissioned by the Government of Canada to 
recommend measures to make the delivery of grant and contribution Programs more efficient and less burdensome, 
while ensuring greater accountability. Sources: http://www.tbs-sct.gc. ca/media/nr-cp/2006/0606-eng.asp#bkl 
http://www.tbs-sct.gc.ca/gcr-esc/docs/2008/ragcp-rapsc-eng.pdf 

37 Sources: http://www.tbs-sct.gc.ca/media/nr-cp/2006/0606-eng.asp#bkl; and. Project Results Report template 
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o Systems could be put in place to better support the sharing of information within 
the Program and with key partners and stakeholders. 

• Ensure project officers are experts in the subject matter area (i.e. understanding the 
recipient base and their achievements). This is hampered somewhat, as noted by a few 
key internal key informants, by the high level of project officer turnover. 

o Systems could be put in place to better support knowledge transfer within the 
Program. 

• Address the limitations imposed by the CSGC with respect to the difficulties in data 
tracking and extraction 38 in terms of timeliness, completeness 39 , usefulness and quality 
of the data. 

• Enhance communications between the Program and funding applicants to ensure they 
are well informed (in a timely fashion) about modifications to the criteria, changes in 
proposal evaluation timelines and, in particular, of the reasons for funding being 
declined. 

• Provide a stable, single point of contact to organizations that can provide support 
across all phases of the proposal development and for post-decision follow-up which 
would in turn help to address previously noted communications issues. 

The documentation review, supported by some key internal informant interviews, noted 
that measures are being undertaken to address some of the areas identified above. 
This includes revisiting Program performance measurement systems, creating an Access 
database (under development) which will enable ongoing project status tracking, and 
(as noted earlier) trying to standardize reporting and making this reporting consistent 
across the regions while taking advantage of the Blue Ribbon panel initiative. 



3S When data is needed, managers make the formal request to CSGC Reports and Outputs, Grants, Contributions and 
Third Party Solutions Transaction Processing Portfolio of HRSDC. As such, it would appear that the Program is 
highly dependent on this Group to obtain data required to inform decision making and assist in program monitoring. 

39 Project officers are responsible for entering performance measurement information/monitoring data into the CSGC 
based on their interpretation of proposals received. While improvements have been noted from 2006-07 to 2007-08 
in data entry i.e. less blanks/missing information (which could be partly attributable to the more rigorous application 
process in collecting such information from recipients in a more systematic way) data is not being collected/entered 
into the system consistently in key areas (“venture",” issue”, “stream”, “activity” and “target audience”). These areas 
are critical to assessing the extent to which the Program is funding projects that are aligned with priorities (its own 
and those of government), meet the needs of the literacy and essential skills community, and will have the greatest 
impact on literacy and essential skills in Canada - particularly in relation to the Canadian workplace. 
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3.2 Evaluation Issue: Performance Monitoring and 
Measurement 

Is an adequate performance measurement mechanism and system in 
place? How could performance measurement be improved? 

Adequacy of performance measurement mechanism and systems: Efforts have been 
ongoing during the evaluation period to further revise, validate and update the 
performance measurement and evaluation frameworks for OLES (which includes the 
component for ALLESP) as well as its reporting regime to more effectively capture data 
that can measure success of funded projects. 

To this end, the Core Funded Organizations 2008-2009 Performance Report and 
Environmental Scan template (APR) was finalized in December 2008 and provided to 
organizations. The Results Reporting templates (for both grant and contribution 
agreements) (PRR) were finalized and implemented as of April 2009. Work is ongoing to 
develop the performance measurement and evaluation frameworks for OLES. Both the 
APR and PRR templates are described in detail below. 

Core Funding 2008-2009 Performance Report and Environmental Scan 

To demonstrate accountability, core funded organizations are required to report on 
overall performance by critically analyzing how well the organization is managed, what it 
has achieved and plans to achieve based on detailed instructions outlined in the APR 
template. Within this document, core funded organizations are asked to provide the following 
information: 

• Basic profile and budgetary information on the organization; 

• A self-assessment of their performance in terms of how it is managed in the following 
seven areas: 

o Governance 
o Results and performance 
o Risk 
o People 

• Actions to be taken to close gaps in performance among the seven areas noted above; 

• A review of performance in terms of what it plans to achieve (outputs and outcomes); 

• Human and financial resource issues, accommodations, infrastructure, etc.; and 

• An environmental scan, which is expected to provide details on the state of LES in 
their particular jurisdiction or areas of expertise. The environmental scan will be used 
by OLES and will be shared with stakeholders to inform and focus policies and 
programs that better address the LES needs of Canadians. 



o Stewardship 
o Client-focused service 
o Accountability 
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The APR was developed in consultation with core funding organizations, national 
organizations and Anglophone and Francophone coalition groups, with revisions to the 
reporting requirements made based on suggested changes. In addition, organizations are 
free to contact OLES should they have any questions or concerns and are encouraged to 
submit the first draft of their reports for additional guidance. 

A total of six national organizations and 16 provincial and territorial coalitions are 
currently required to complete the APR. The Program will work with core funded 
organizations for up to a year in advance of the end of their agreements to complete the 
APR in a way that accurately reflects their work. The information provided in the APR is 
then analyzed and discussed within the Performance Measurement and Intergovernmental 
Relations (PIR) and Program Operations group, and ultimately used as the basis for the 
negotiation of the next agreement with these organizations. 

It is clear that the new reporting templates are improvements, especially when compared 
to the previous reporting regime, and will help to capture data and information that can 
measure success of funded projects. As OLES moves forward with this template, the 
evaluation team noted the following: 

• A number of the questions asked of core funding recipients are quite detailed and 
labour intensive. Given that the APR is targeted towards all core funded organizations, 
irrespective of size, it is possible that smaller organizations (i.e. those with only a few 
core staff) may not have the resources or data collection capacity to follow through 
with the reporting requirements. 

• The APR contains several double-barrelled questions (i.e. questions with more than 
one question embedded within) which may make it difficult for some organizations to 
ascertain which issue they ought to respond to. 

• The APR contains “Yes” or “No” questions which do not provide adequate information 
on outcomes achieved. 

Finally, given the administrative challenges already noted, questions arise as to whether 
the Program would have sufficient time or the capacity to compile, review and report on 
such broad and detailed data in order to successfully measure the performance of funded 
organizations. 

Project Results Report / Project Results Report for Grant Agreements 

The PRR has been designed to respond to the recommendations of the Blue Ribbon panel 
on Grants and Contributions, particularly to strea ml ine reporting and improve accountability 
throughout the lifetime of a project. The PRR will replace the various reports currently 
used through the year and at project closure. As previously noted, only new projects and 
current grants and contribution recipients (those signed April 1, 2009 and later) have 
begun reporting via the PRR. 
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The PRRs were developed based on an examination of the different types of templates 
that recipients were previously asked to fill out throughout the life cycle of a project. 
They were streamlined to include a) those parts that are required to be reported on under 
grants and contributions and b) that help projects show results, rather than just their 
activities. 

The objective of the new PRR is to provide an understanding of the project’s overall 
performance by critically analyzing what is planned to be achieved and what has been 
achieved (monitoring and evaluation), rather than just describing the activities they are 
carrying out. The PRR summarizes: 

• the expected and actual results of the project 

• activities and outputs that will be carried out to achieve the results 

• possible issues which could affect the success of the project, and 

• the progress of the project. 

In doing so, the funded applicants are expected to provide information on the quality of 
the project’s outputs, the outputs’ contributions to expected project results, the achievement 
of expected project results, lessons learned and success stories, the dissemination 
strategy, and project impacts. Results are reported based on the following timing schedule: 
quarterly updates (activity and output status); a review of project outputs; final project 
review (project closure); and project impact assessment (6 to 12 months after project 
closure). Funding recipients are expected to revise the PRR accordingly as the project 
progresses and any key factors change. 

Given that the survey of funding applicants does not include organizations who applied 
for funding after the 2007-08 fiscal year, the evaluation was unable to gather the views of 
funding recipients regarding the new reporting requirements. However, it would be worth 
keeping in mind some of the issues raised by key informants and well as survey participants 
concerning the previous reporting requirements, as OLES moves forward with the new 
PRR. These issues include: 

• The resource realities and limited data collection capacity which exists within many 
organizations, especially smaller ones. It is acknowledged that this ought to be less of a 
burden, considering that previously, OLES required monthly reports (as compared to 
the PRR’s quarterly reports). 

• While OLES held a full-day workshop to explain the new reporting tools to literacy 
coalitions and national organizations, current grant and contribution recipients who 
were not involved in this workshop may require ongoing efforts on the part of OLES to 
ensure buy-in. 

• Again, while a great deal of project information will be collected, previous concerns 
raised by key informants, as well as the administrative data review, regarding OLES’ 
ability to capture, roll-up and interpret the resulting data remain. 
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Accuracy: 



Extent to which the performance indicators accurately reflect outputs and results: 
From a review of the performance measurement strategy, there appears to be an appropriate 
balance between quantitative and qualitative indicators. However, a few internal key 
informants noted that the nature of monitoring at this time is solely based on ‘output 
monitoring’, particularly on the number, value and duration of grants and contributions 
projects distributed by province, stream, target group, activity area, issue area and venture 
i.e. outputs of funded projects. At the same time, information on the latter five codes is 
seriously incomplete within the CSGC, making it virtually impossible to assess the 
Program on these factors. 

As evidenced by the administrative data analysis, data is not collected in enough detail 
within the CSGC to truly inform the Program in terms of achievement of expected 
outputs and outcomes. As previously noted, many of the fields of analysis and codes for 
“stream”, “activity”, “issue” etc. are very broad in scope and do not provide enough detail 
as to the nature of projects (e.g. one possibility for project activity was “knowledge”). 
At the same time, the data is not inputted consistently for these data categories and 
therefore this information is not available for the majority of projects funded. In addition, 
there is no evidence that the Program is collecting information to measure or monitor its 
own performance in terms of program and service delivery. It should be noted that 
additional data collection measures noted above are expected to ensure that performance 
information and data is collected and reported in a more consistent and complete fashion. 

Availability: 

Extent to which performance information and data can be collected: As evidenced by 
the file review, the collection of performance information and data is partially dependent 
upon the extent to which recipients include this information as part of proposals and 
reporting. As noted through the documentation review and file review findings, the 
preparation of proposals requires more supporting research and more detailed plans 
including action plans. The introduction of checklists, as part of the CFP Application 
Guide and Kit are expected to assist applicants in ensuring key forms, documents and 
mandatory information requirements are met as part of their proposals while providing 
guidance for future reporting. Finally, applicants are expected to include as part of their 
proposals a detailed evaluation framework. This framework must include information on 
the indicators and methodologies that will be used to evaluate the success of the project, 
the resources required, expected deliverables and target groups. 

This, along with the introduction of the PRR are intended to assist funding recipients and 
thus the Program in ensuring performance information and data is collected and reported 
in a more consistent and complete fashion. In addition, the environmental scan will be 
used by the OLES, and will be shared with stakeholders, to inform and focus policies and 
programs that better address the LES needs of Canadians. 

The survey of funding applicants reveals, while results monitoring or data collection is 
fairly pervasive among funded respondents (79% claim to undertake these types of 
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activities), any effort by ALLESP to gain a detailed understanding of the “users of material 
outputs” produced by the projects it funds would prove quite challenging. Half of funded 
respondents do not track who uses their material output and among those who do this 
activity, approaches are varied and in many cases very informal. 

Quality: 

Extent to which the performance data being collected is accurate and complete: Project 
officers are responsible for entering performance measurement information and monitoring 
data into the CSGC based on their interpretation of proposals received. While improvements 
have been noted from 2006-07 to 2007-08 in data entry i.e. less blanks or missing 
information, a high proportion of data is not being collected or entered consistently into 
the system i.e. fields are left blank, specifically for the following key areas: “stream”, 
“activity”, “issue”, “venture”, “activity” and “target audience” as evidenced through the 
administrative data analysis. 40 In addition, the information that is available is significantly 
weighted towards the province of Quebec. The one-year protocol agreement is a major 
factor behind this observation. The protocol agreement leaves the administration of 
project funding disbursement to Quebec, which is inclined to support much smaller 
projects than the ALLESP norm. Although Quebec administers the protocol, projects are 
still given final approval by the Minister of HRSDC. Therefore, data that is received for 
project funding is complete and up-to-date as it must first go through the approval process 
instituted in Quebec. 

In addition, a number of entries for “issue” and “venture” code over the two-year period 
were identified by ‘NA-not applicable’. The ‘NA-not applicable’ classification for projects, 
similar to previous years, seems to continue to be a ‘catch-all’ for those projects where 
project officers may have alternatively identified or chosen more than one issue category 
or venture code. This makes it extremely difficult to examine program-related data in any 
depth or how trends change over time. 

Lurthermore, a number of entries for “target audience” were identified as “other” making 
it difficult to fully articulate all targets groups of project funding. 

Linally, the project database review conducted for the evaluation can account for only 
$35,238,597 in ALLESP grants and contributions spending between 2006-07 and 2007-08, 
which is approximately $13M less than $48. 1M spent according to the Chief Linancial 
Officer Branch of HRSDC. 

Usefulness: 

Extent to which the performance data supports decision-making and departmental 
accountability requirements; Perceived usefulness of performance data and reporting: 
Other than informal management reviews, performance information does not appear to be 
used to inform management decision-making processes. As previously noted, limited 
project results and outcome information was collected during the period under evaluation 



40 Please refer to Tables 10 through 13 for a detailed breakdown. 
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and thus being analyzed to support decision-making (the CSGC is primarily a database 
driven to support financial accountability for grants and contributions and not outcomes 
assessment). It should be noted that the Program has demonstrated that it has been 
monitoring and assessing the CFP and decision-making processes and has attempted to 
make changes in order to ensure they are more efficient and effective. 

Overall, the review of program documentation supported by some internal key informant 
interviews identified the need for more effective accountability for results and for a 
higher standard of planning and performance management. Key weaknesses identified 
include few meaningful targets against which to compare results achieved, inconsistent 
and decoupled planning, monitoring and reporting at all levels of OLES, no systematic 
processes or systems for gathering performance information, and the inability to link 
resources to results. 

OLES has identified the need to develop an integrated framework to support its drive 
towards managing for results. This includes integrating planning and actions to achieve 
clearly-defined outcomes and establishing effective monitoring and reporting on 
performance. This reflects OLES’, and more broadly the Branch’s and Department’s, 
integrated approach to planning, decision-making, activities and reporting to support more 
effective delivery on plans and priorities. An important part of this work is delivering on an 
effective performance measurement regime that supports internal decision-making while 
responding to a number of external reporting imperatives. To support this work, the OLES 
Management Team has dedicated resources to implement a comprehensive and structured 
plan to ensure the Program is “managing for results” (as discussed above). 

Ways in which the performance measurement mechanism and systems could be 
improved: According to most internal key informants, an adequate performance system 
does not exist for collecting performance information, in particular, outcome information 
(i.e., intermediate and long-term) both on projects as well as the Program. A few internal 
key informants noted that these challenges are partly attributed to the fact that ALLESP 
has “inherited” three previous programs with their own respective procedures and systems 
for data collection. As such, it has taken some time to develop an integrated approach and 
means to better collect information for both projects and the overall Program. 

As previously noted, the collection of performance information and data is highly 
dependent upon the extent to which recipients include this information as part of 
proposals and reporting; however, earlier findings from key informant interviews suggest 
that the nature of monitoring at this time is solely ‘output monitoring’. During the 
evaluation period, limited performance information was being collected to support 
outcome monitoring due to limited follow up at the end of funded projects. 

Additional ways in which performance measurement systems can be improved, as 
suggested by internal key informants, include the need for: 

• better understanding and development of performance measures with staff at the 
beginning of the fiscal year to ensure data capture and refinement; 
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• outcomes-based focus to measurement and electronic reporting to decrease administrative 
burden and allow for better information integration; 

• contact and communication with partners to identify gaps in information; and 

• more rigour in the initial assessment of proposals and more openness and responsiveness 
about the changes that are needed to ensure eligibility. 

Is the basic information necessary for the summative evaluation being 
collected? 



Information on Funding Applicants and Recipients 

As evidenced through the administrative data analysis, the tombstone data captured on 
funded applicants and recipients appears to be 100% complete in the CSGC. This includes: 

• Organization name; 

• Organizational Type or Category (i.e. not-for-profit, public or private sector); 

• Mailing Address (including city, province, postal code); 

• Other contact details such as organizational contact name, language, telephone number 
and e-mail where available; and 

• Organizational mandate. 

Conversely, the data captured on unfunded applicants is limited. Besides organization 
and primary decision maker contact information, ALLESP has little to no information on 
unfunded applicants and the projects for which they had applied for funding. 

Information on Project Outputs 

There are 19 Venture codes that capture the outputs of funded projects that range from best 
practices, to educational and resource material, to research studies, to volunteer development. 
As noted earlier a review of the CSGC reveals that, in general, a high proportion of data for 
“Venture” is not being inputted into the system i.e. either being left blank (106 of 288 entries 
or 37%) or identified by ‘NA-not applicable’ (36 of 288 entries or 12.5%). 

Profiles of the Target Groups 

As noted earlier in this report, ALLESP continues to experience difficulties in its ability 
to identify information on target groups, as well as consistently tracking this data as part 
of project reporting and within the CSGC. The CSGC currently allows for the tracking of 
25 different “target groups”. Despite this capacity, over 69% of the information on the 
Target Audience or Group was missing or blank in the CSGC. 

OLES does not collect detailed information on end-users or target groups (such as the 
profile of target groups; number of individuals having completed their learning activity; 
etc.) since, according to the evaluation of the OLT, the collection of such data was not 
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feasible. As such it was recommended that information on end-users be limited to 
tracking data for the purposes of generating a target group profile. This information is 
now expected to be provided by funding recipients as part of their closeout reports, the 
PRR and the APR. 

3.3 Evaluation Issue: Preliminary Findings - outputs 
oriented 

What type of Funding Agreements have been entered into under ALLESP? 

Type of Funding Agreements entered into under ALLESP: Evidence collected through 
the large majority of internal key informants interviews and the administrative data 
analysis noted that the primary funding mechanism used under the Program is grants- 
based. Over 266 or 92% of projects funded over the 2006-07 and 2007-08 fiscal years 
were grant-based (145 or 94 % for 2006-07; 121 or 90% for 2007/08) with the remaining 
funding source being contribution-based (22 or 8%). As previously noted, one of the key 
recommendations of the previous NLS evaluation was to move away from grant-based 
funding. 



Table 9 

Total Project Funding (All Sources) by Fiscal Year 41 


Funding Type 


(2006-07) 


(2007-08) 




# of 

projects 


Funding 

Amount 


# of 

projects 


Funding 

Amount 


ALLESP Contribution 


9 


$6,089,698 


13 


$5,850,210 


ALLESP Grant 


145 


$18,704,578 


121 


$14,120,919 


Total 


154 


$24,794,276 


134 


$19,971,129 



Given the high proportion of data that is not being collected and/or entered consistently 
into the system it is difficult to complete a full analysis of the type of projects that are 
being funded (including the area funded, the tools and outputs). As such, it is difficult to 
fully evaluate the extent to which the Program is funding projects that are aligned with 
priorities (its own and those of government), meet the needs of the LES community, and 
will have the greatest impact on LES of individuals in Canada - particularly in relation to 
the Canadian workplace. In addition, as previously noted, the data that is available seems 
to be heavily weighted towards the province of Quebec. Nevertheless, of those 86 out of 
288 projects for which information was available, 59 (68%) focused on communities, 
15 (17%) on families and 12 (14%) on the workplace during the evaluation period. 



41 An additional project was funded under the ALLESP, however the CSGC does not designate whether it is a grant or 
contribution. Program confirmation has noted that this project is a contribution. 
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Table 10 

Distribution of Projects by Stream Code by Fiscal Year 


Stream Code 


(2006-07) 


(2007-08) 


Total 


Communities 


8 


51 


59 


Families 




15 


15 


Workplace 


2 


10 


12 


(Blank) 


144 


58 


202 


Total 


154 


134 


288 



Within the four key eligible areas of activity, the administrative data analysis indicates 
that of those 86 out of 288 projects for which information was available, 43 (50%) of the 
projects focused on addressing “strengthening capacity”, followed by 25 (29%) on 
“promotion”, 11 (13%) on “knowledge” and 7 (8%) on “innovative approaches”. 



Table 11 

Distribution of Projects by Activity Code by Fiscal Year 


Activity Code 


(2006-07) 


(2007-08) 


Total 


Knowledge 


1 


10 


11 


Strengthen Capacity 


7 


36 


43 


Promotion 


- 


25 


25 


Innovative Approaches 


2 


5 


7 


(Blank) 


144 


58 


202 


Total 


154 


134 


288 



These findings would suggest that more work may be needed to increase awareness and 
understanding of the importance of LES, while issuing more calls relevant to the other 
three eligible activity areas, (in particular “knowledge” and “innovative approaches”) so 
as to remain relevant to the Program’s stated objectives and desired outcomes. These 
efforts in turn would help to address some of the key needs and priorities identified 
within the LES community to raise awareness and understanding, and increase research 
and innovation in the LES area. 

Only 120 out of 288 projects in the CSGC contained information on the types of issues to 
be addressed. Of these, the largest proportion were identified as having addressed “basic 
skills and literacy”, particularly in 2007-08 despite the addition of a new category to the 
system for “essential skills ”. 42 While the addition of this new category may be reflective 
of Program directions to integrate LES, the high incidence of the use of “basic skills and 
literacy” to classify projects may indicate difficulties in data tracking (officers 
interpretation of proposals) or that, in fact, ALLESP has not significantly shifted funding 
towards LES in relation to labour market participation. This latter observation may reflect 
the Program’s continuing challenge to encourage applicant organizations to expand their 
work beyond literacy to include essential skills in relation to labour market participation. 



4 ’ Table 12: Distribution of Projects by Issue Code by Fiscal Year. 
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Table 12 

Distribution of Projects by Issue Code by Fiscal Year 


Issue Code 


(2006-07) 


(2007-08) 


Total 


Basic skills and literacy 


2 


49 


51 


Computer Assisted / Managed Instruction 


5 




5 


Essential Skills 




4 


4 


Family 


6 


3 


9 


Health Care 


1 


2 


3 


Labour / Business 


2 




2 


Learning opportunities 




1 


1 


Network development 




2 


2 


Numeracy / Mathematic 




1 


1 


Official Language Minority - French 




7 


7 


Partnership development 


2 




2 


Plain Language 


1 


1 


2 


Preventative 


2 




2 


Resources Centres / Clearing Houses 


1 




1 


Teaching Methods / Practitioner Training 


7 


12 


19 


Workplace 


5 


4 


9 


Not Applicable 


43 


14 


57 


(blank) 


77 


34 


111 


Total 


154 


134 


288 



Of those 146 out of 288 projects for which information was provided 43 , key project outputs 
most frequently noted (i.e. 8 or more projects) and further displaying consistency over the 
two years, included Education and Resource Material; Promotions and Publications; 
Research and Studies; Training and Skills Development; Conferences, Meetings, Seminars 
and Workshops; and Demonstrations and Pilot projects. 

Extent to which the development of literacy & essential skills research, tools and 
supports through ALLESP funding as well as the Program’s support of provincial 
and territorial coalitions & national organizations, contributed to, or is expected to 
contribute to ALLESP’s immediate outcomes: In terms of whether results will be 
achieved, a few internal key informants noted that it is too early in the Program lifecycle 
to answer this. This was impacted by the fact that there were some projects that had not 
yet ended, as well as the difficulties being experienced with measurement and attribution 
(i.e. demonstrating achievement of project results). Of the 288 funded projects (Table 13), 
146 contained information on project outputs. 44 Of these, the most frequently identified 
(i.e. 8 or more projects) included Education and Resource Material; Promotions and 
Publications; Research and Studies; Training and Skills Development; Conferences, 
Meetings, Seminars and Workshops; and Demonstrations and Pilot projects. 



43 Of the 146 projects, 49 were for the province of Quebec 

44 Of the 146 projects, 49 were for the province of Quebec 
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Table 13 

Distribution of Projects by Venture Code by Fiscal Year 


Venture Code 


(2006-07) 


(2007-08) 


Total 


Best Practices 




2 


2 


Conferences / Meetings / Seminars 


1 


2 


3 


Conferences / Meetings / Seminars / Workshops 


12 


6 


18 


Demonstration / Pilot Projects 


8 


10 


18 


Educational / Resource Material 


18 


9 


27 


Endowments 


1 


- 


1 


Evaluation 


2 


- 


2 


Film / Video 


- 


1 


1 


Innovation 


- 


1 


1 


Learning Events 


- 


1 


1 


Learning models 


- 


2 


2 


Partnership 


1 


2 


3 


Promotions 


5 


1 


6 


Promotions / publications 


- 


20 


20 


Publications 


1 


- 


1 


Research / Studies 


8 


11 


19 


Training / Skill Development 


8 


11 


19 


Volunteer Development 


- 


2 


2 


NA - not applicable 


16 


20 


36 


Blank 


73 


33 


106 


Total 


154 


134 


288 



Some internal and external key informants noted that without Program funding the LES 
community would not have the broad awareness it has of various LES issues, nor would 
it have been able to develop the products (i.e. programming, tools) supported by 
ALLESP-funding. This is also demonstrated by the results of the survey of applicants. 
While recognizing the possibility of bias on the part of respondents, the survey indicates 
that the vast majority of projects would not have been undertaken in the absence of 
ALLESP funding. When asked what effect not receiving ALLESP funding had on their 
proposed projects, nearly half (45%) of unfunded respondents cancelled the project and 
another 29% deferred the project until other funding could be found. Conversely, when 
funded respondents were asked what would have happened if they had not received 
ALLESP funding, nearly all believe their project would have either been cancelled 
outright (33%) or it would have been deferred until other funding could be found (60%). 

The majority of funded respondents believe their project is, or will be “successful” (76%), 
or “somewhat successful” (21%) in achieving its objectives. When asked what factors 
contributed most to the success of their project, respondents most often reported that it 
was the receipt of ALLESP funding itself (93%). In addition, respondents reported that 
their projects will produce a wide variety of outputs, most often citing “awareness raising 
or promotional events” and “training tools, models and materials”, and that these outputs 
typically benefit a wide variety of organizations. Local organizations and non-profit 
service organizations are the most likely targets of material outputs produced by funding 
applicants. 
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In addition, the survey results do provide some insight into the perceived importance of 
ALLESP funding in terms of its contribution to certain key project outcomes and outputs, 
benefiting a wide array of organizations. In particular, funded respondents expected their 
projects to contribute to a wide host of possible outcomes, the most common consisting 
of “greater awareness of literacy and essential skills programs, services and/or issues” 
and to the “develop [ment] of best practices tailored to the particular needs of [their] target 
audience”. 

The evaluation examined to what extent the ALLESP has achieved or will achieve its 
immediate outcomes. The results are as follows: 

• improved dissemination, transfer and application of knowledge and information 

At this time, according to a few external key informants, there is limited awareness as to 
what projects are being funded by ALLESP. In particular, given that OLES does not 
publish a list of projects that are funded, their outputs (i.e. tools, research, etc.) and 
outcomes, the quality of the work funded and the extent to which tools can be used more 
broadly in the LES community is unknown. While a few internal key informants noted 
that the Program hosts an ES website that is targeted towards key groups, it was 
acknowledged that it was difficult to determine who the research or documentation is 
targeted towards and how to inform these groups of this material. It is important to note 
that this website is not for ALLESP or OLES but rather for the Essential Skills Research 
project and ES Profiles. At the same time, a few internal key informants indicated that 
dissemination is a weak point of funded projects and is something done ‘after the fact’ 
and often undertaken in a passive way (i.e., through websites) as opposed to investing 
project resources in targeted outreach and distribution. 

Without effective communication of project outputs and outcomes, and monitoring of 
dissemination and quality of outputs, the impact of the Program funded knowledge, tools 
and information cannot be assessed. As noted by a few internal key informants, the hope 
is that by funding literacy coalitions and organizations, they will be prompted to take 
some responsibility for dissemination, especially given their pre-existing networks with a 
variety of LES stakeholders. 

In addition to difficulties with dissemination and transfer of knowledge noted above, 
some external key informants noted that Canada does not have enough research and 
development in the LES area (for example in areas such as literacy and the aging 
population, ESL/FSL, essential skills in the workplace). As a result, more resources 
(i.e. funding) are needed to support this type of research. In addition, it was noted that 
there is not enough comparative research between provinces and territories as well as 
linking to what’s happening internationally (such as the UK, US, Australia). 

As such, Program efforts to improve the dissemination, transfer and application of 
knowledge and information seem limited at this time. According to a few internal key 
informants the new role for policy and planning is to gather and analyze information from 
funded projects on what has been learned, what is happening on the ground and determine 
where to focus next should help overcome some of these challenges. Furthermore, more 
rigorous requirements as part of the application process (i.e. funding proposal requirements 
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as well as the introduction of a Project Results Report template) should assist efforts to 
improve dissemination and transfer of knowledge from funded projects. 

• increased capacity of funding recipients and other stakeholders 

While it was recognized by a few internal and external key informants that the Program is 
helping to strengthen the means organizations have to do their work, it was noted that it is 
difficult to assess the extent to which the Program is successfully building capacity 
within the LES community based on the limited information available. In the short-term, 
according to a few internal key informants, Program funding is stretching the capacity of 
recipients; however, once funding stops the benefits and often the ability to continue the 
project stops. Funded organizations seem to have the continued need for funding to 
support ongoing delivery. Furthermore, given the transient nature of the FES community’s 
workforce (voluntary-based), staff turnover tends to be high, impacting knowledge and 
expertise. It was suggested by one internal key informant that AFFESP should continue 
to work with funding applicants to examine the organization’s own capacities to deliver 
on expected results and to determine the correct level of funding to support the proposed 
project’s objectives. 

Key informants provided limited feedback on the Program’s support for provincial and 
territorial coalitions, other than to note that coalitions receive core-funding to carry out 
specific responsibilities. This funding provides these organizations with stability to carry 
out community work and also satisfy requirements from AFFESP/OFES (i.e. submit a 
proposal every couple of years). Furthermore, these key informants noted that uncertainties 
exist within the FES community around the amount of funding in national organizations 
and the extent to which it is getting to the people that need it most to increase their 
skill levels. 

• increased awareness of the benefits of and opportunities for adult learning, literacy and 
essential skills 

While there is a high level of acceptance of the context and challenges, a few key informants 
noted that more work needs to be done in this area to increase awareness and understanding 
of the importance of FES, particularly within the workplace. It was recommended by a 
few internal key informants that the federal government consider investing in a concerted 
media campaign similar to that which was developed for ‘anti-smoking awareness’. 
Such a campaign would require national coordination amongst the various levels of 
government as well as with the FES community. 
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4. Key Conclusions and Recommendations 



The overall objective of the formative evaluation of the Adult Learning Literacy and 
Essential Skills Program (ALLESP) was to examine a number of key issues and questions 
pertaining to program design and delivery, performance monitoring and measurement and 
preliminary findings. 

This section contains the key conclusions developed from the findings of the formative 
evaluation of ALLESP followed by some recommendations for the Office of Literacy and 
Essential Skills (OLES) to consider. 



4.1 Evaluation Issue: Program Design and Delivery 

Program Objectives and Activities 

While the objective of the ALLESP is considered valid at a high level, it is viewed as 
being too broad, far reaching and perhaps not as targeted as it could be. Specifically, the 
Program’s ability to contribute to facilitating u the creation of opportunities for Canadians 
to acquire the learning, literacy and essential skills they need to participate in a knowledge 
based economy and society” is viewed as being hampered by a number of factors 
including the fact that the Program does not actually fund direct programming, limited 
Program funding relative to the size of the issue, and delayed funding decisions. 

An assessment of the Program logic model 45 revealed a number of inconsistencies that 
may not reflect the current Program context or directions. In particular, the word 
‘ opportunities’ may lead to the mistaken belief amongst funding applicants that ALLESP 
is undertaking program delivery, when in fact this statement is viewed as being “more 
about leveraging opportunity” to support the development of more effective LES 
programs and tools and ultimately the increased acquisition of LES. In terms of the 
Program’s aim to “ promote lifelong learning by reducing non-financial barriers to adult 
learning” this evaluation revealed that there is limited evidence that the Program is really 
‘ reducing non-financial barriers’ . Rather it was noted that the objective or focus of the 
Program is about developing knowledge and building expertise as to what works in 
building LES for Canadians to participate in a knowledge-based economy. Critical to this 
development is the dissemination of knowledge, tools and research created from 
ALLESP-funded projects so that it can be shared, applied and replicated or adapted 
elsewhere. The evaluation also noted that there is less emphasis on ‘ adult learning ’ and 
more emphasis on building the LES that Canadian adults need for work, learning and life. 
As a result, the name of the Program was questioned and the use of ‘ adult learning’ in 
defining the Program objectives and its use in each of its key activities. 



45 It should be noted that the logic model examined as part of this evaluation was deemed a “work in progress”. 
A revised version is currently being developed for the Program based on the new model developed for OLES to 
ensure linkages between the two. 
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While four streams of activity (research and knowledge; capacity-building; innovative 
tools and practices; and promotion and awareness) are identified in the Program logic 
model, evaluation findings suggest that a large majority of Program resources are 
targeted towards funding projects aimed at ‘ strengthening capacity ’. 46 

It is recommended that Program objectives and activities should be revised to more 
accurately reflect the direction and scope of the Program and to ensure outcomes are 
achievable and measurable. 

These efforts should be undertaken with the recognition that intermediate and ultimate 
outcomes, by definition, are acted upon by multiple players and factors and that the 
Program is not involved in direct delivery. The Program should attempt to prioritize 
each of its key activities and determine the extent of resources to be dedicated to each. 
This should then be reflected in the Calls for Proposals (CFPs) to ensure that potential 
applicants understand the range of activities that ALLESP will fund. 

Challenges of Program Realignment 

The evaluation revealed that ongoing issues related to the program integration process 
have impacted program co-ordination and programming. From a departmental perspective, 
the alignment of the three previous programs under the ALLESP was necessary in order 
to improve program impacts on LES and to reduce duplication. External stakeholders 
overall were unclear as to why a Program re-alignment was needed. In fact, re-alignment 
was viewed as taking away from the effectiveness of previous programming, resulting in 
minimal improvements and a marked lack of ‘field presence’ of ALLESP in the regions. 
In addition, a weakening of partnerships and collaboration was noted as a result of program- 
re-alignment, particularly within the provinces and territories, who view themselves as being 
designated as “stakeholders” under the ALLESP, rather than full and equal partners as 
they felt they were treated under the National Literacy Secretariat (NLS). 

This evaluation also found little evidence that such a re-alignment has improved efficiency 
in the overall administration of the Program as noted by delays in the CFP process, and 
the views of funding applicants, who were less than satisfied with a number of aspects of 
the application, administration and follow-up process. In particular, delays in CFPs and 
funding decisions have resulted in waning project momentum and also compressed 
timelines for project delivery, and ultimately frustration and declining interest among 
partners, sometimes resulting in partner drop out. Furthermore, program re-alignment has 
led to the impression of more insular approaches to Program delivery and an overall 
decline in service, especially amongst applicants who had experience with the three 
previous programs. 

While efforts to broaden the scope of funding recipients to include employers, academia, 
unions, etc. have been noted, a large portion of ALLESP funding continues to be directed 
towards the not-for-profit sector. The evaluation also noted that the primary funding 



46 It should be cautioned that these findings may be reflective of challenges noted throughout this report with the 
tracking of this type of information in the Common System for Grants and Contributions (CSGC). Without a 
complete database it is difficult to draw any definitive conclusions in this area. 
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mechanism used under the Program continues to be grants-based. Given the recommendations 
of previous NLS evaluations to move away from grants, these findings would indicate 
that ALLESP has yet to achieve this objective. 

Despite the above-noted challenges, evaluation evidence indicates that the programs are 
now becoming more integrated and while there appears to be separate cultures between 
essential skills and literacy, findings from a few key internal informant interviews note 
that OLES is making a concerted effort to integrate these cultures under one organizational 
framework. Progress has also been made in some areas including building awareness 
about ALLESP’ s focus on LES in relation to labour force participation and in rebuilding 
partnerships and communications with provincial and territorial partners in priority 
setting. More work is needed to ensure that the various components work together while 
improving openness, transparency and clarity in communications, re-building partnerships 
and collaboration and addressing administration efficiencies. 

It is recommended that the Program continue its efforts to communicate, support and 
consult with stakeholders and funding applicants in order to ensure openness and 
transparency with regards to its priority setting and decision-making. 

Moving forward it will be important for the Program to ease uncertainties within the 
LES community through ongoing communication and outreach on the benefits of 
aligning proposed project approaches with ALLESP’s key priorities - in particular 
the importance of investing in LES within an economic context while not dispelling 
the importance of strengthening LES to support social inclusion. The Program should 
also continue to ensure clarity in communications and decision-making to eliminate 
any perceptions of an unwillingness to share information as well as confusion over 
how priorities are set. 

Given the decline among smaller provinces (mostly Atlantic) and smaller organizations 
in the share of overall ALLESP funding and number of projects funded, efforts should 
be made to support those organizations with more limited capacity as compared to 
larger-scaled and national organizations. The Program may also wish to consult with 
representatives from the identified areas and organizations to determine if more 
supports and/or communications are required to help support their application process. 

As part of the CFP process, the Program should improve communications and 
consultations with: 

• unfunded applicants regarding the CFP and decision-making process given that 
the changes the Program has implemented do not seem to have increased 
satisfaction with the process or the explanation given for the funding decision. 
As part of this process, the Program should review the effectiveness of its new 
approach to informing applicants when their proposal was not approved for 
funding and do more targeted follow up with clients to ensure that the changes are 
achieving their intended results or whether further improvements are required. 

• both funded and unfunded applicants need to be engaged following each proposal 
process. Post-CFP satisfaction surveys could be conducted to discuss satisfaction 
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with proposal process, the level of support provided and any suggestions for 
improvement. OLES may also wish to consider developing service standards for 
CFPs and grants and contributions administration. 

It is widely recognized that governments alone cannot resolve a complex policy area 
such as LES which is influenced by a number of social, economic and political factors. 
Efforts need to continue to recapture the trust and partnerships, especially between 
the two orders of government with more effective and open communication and 
outreach. For example, it was suggested by a few internal and external key 
informants that ALLESP may want to consider re-instating the provincial review of 
proposals in order to provide provinces with an opportunity to read proposals and 
‘stay in the loop’ or to consider re-instating an annual meeting of federal and 
provincial LES representatives. 



4.2 Evaluation Issue: Performance Monitoring and 
Measurement 

Performance Measurement and Data Collection 

This evaluation has found that while a performance measurement strategy and indicators 
were identified for the Program, efforts have been ongoing during the evaluation period 
to further revise, validate and update the performance measurement and evaluation 
frameworks for OLES, as well as its reporting regime to more effectively capture data 
that can measure success of funded projects. 

A number of reporting templates were introduced during the course of the evaluation. 
These templates are expected to improve the Program’s ability to collect information with 
regards to the quality of project outputs, the achievement of expected outcomes, the 
reporting and dissemination of project results, as well as project impacts. Overall, the 
new reporting templates are improvements, especially when compared to the previous 
reporting requirements, and will help to capture data and information that has not been 
previously collected. As this process moves forward, OLES will have to keep in mind 
some of the issues raised concerning the previous reporting requirements. These issues 
include the possibility that, despite efforts to reduce the reporting requirements, some 
funding recipients, especially smaller organizations, may still perceive the new templates 
as complicated and time-consuming to finish. As such, OLES will have to confirm that 
funding recipients understand the new reporting requirements, while providing continuing 
support to ensure that they have the capacity to complete the templates. At the same time, 
experience with previous reporting has shown that OLES may lack the capacity to 
actually collect, monitor and analyze the data provided by these reports. 

Challenges with the Common System for Grants and Contributions (CSGC) 

Project officers are responsible for entering performance measurement information and 
monitoring data into the CSGC based on their interpretation of proposals received. 
The system is maintained by CSGC Reports and Outputs, Grants, Contributions and 
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Third Party Solutions Transaction Processing Portfolio of HRSDC. The information 
contained in this database is to be used to develop a profile of funding applicants and in 
reviewing program performance, compliance monitoring, management and reporting. 

When examining the data provided in the CSGC, this evaluation revealed that while 
improvements to data entry were noticeable from 2006-07 to 2007-08, a large amount of 
missing data continued to be noted within various codes, including funding stream, 
activity, issue, venture and target audience and group codes. Without complete information 
on these codes, it is very difficult to fully analyze the types of projects that are being 
funded (eg: area funded, tools, outputs, etc.). Therefore, it is difficult to fully evaluate 
the extent to which the Program (1) is funding projects that are aligned with priorities 
(its own and those of government), and (2) meets the needs of the LES community, 
including how it will have the greatest impact on the LES of individuals in Canada, 
particularly in relation to the Canadian workplace. 

It was noted that ALLESP is taking measures to address some of the key challenges 
related to the database including revisiting Program performance measurement systems 
and creating an Access database (under development) which will enable ongoing project 
status tracking. Caution is urged as the Program moves ahead with the two databases. 
This was previously done under the NLS and problems noted included program officers 
not sufficiently keeping up either database or critical variables for tracking Program 
performance, as well as outcomes not being captured. 

It is recommended that OLES continue with its commitment to improve its data 
collection systems so that sufficient and complete information is collected for 
monitoring and reporting purposes. In conjunction, it is recommended that proposal 
and reporting forms be updated to better reflect the CSGC codes and categories, 
thereby easing the burden on Program officers. Finally, recent revisions to the 
Program ’s reporting templates will require ongoing communication and support for 
funding recipients. 

Given the above factors, it is concluded that during the evaluation period of 2006-07 
and 2007-08, information was not being collected, to sufficiently inform the upcoming 
summative evaluation in terms of completeness, quality and accuracy. With the 
development of the detailed templates, updated OLES logic model and associated 
indicators, efforts are underway to improve upon this. Given the current data 
tracking issues highlighted by the administrative data review, and the detailed nature 
of the new reporting templates, OLES will have to work to ensure that the 
information provided by funding recipients is tracked in a systematic manner so that 
it is useful for program performance review, compliance monitoring, management 
and reporting, as well as for the future summative evaluation. 

As such, it is essential that the performance measurement data previously collected, 
including information on funding streams, project activities, issues, outputs, etc. be 
inputted into the CSGC, as well as the new Access database, in order to develop 
accurate profiles of funded projects and targets. The Program should also examine 
ways of refining proposal forms and reporting requirements in order to force 
funded applicants to make a choice among the various possible variables that reflect 
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what already exists in the CSGC. This would require the completion of all fields 
and eliminate the requirement for Program officers from having to interpret 
proposal submissions. At the same time, the Program should revise the database 
codes to ensure that they are relevant to ALLESP and not previous programs. 

In addition, while the evaluation found that funding recipients were satisfied with the 
ALLESP’s reporting requirements, with the introduction of the new APR and PRR 
templates, as well as the increase in use of contribution agreements, OLES will have to 
provide ongoing support to funded organizations to ensure that this trend continues. 
In particular, OLES will need to ensure that the type of engagement that is ongoing 
with organizations responsible for completing the APR templates is replicated for those 
organizations responsible for the PRR templates, while ensuring all funding recipients 
are aware of the related support tools. This will help to improve the buy-in for the new 
reporting requirements, especially amongst smaller organizations. 



4.3 Evaluation Issue: Preliminary Findings - outputs 
oriented 

Dissemination, transfer and application of knowledge and information 

The findings of this evaluation confirm that dissemination and transfer of knowledge 
from funded projects is weak, pointing to a need to further improve the dissemination and 
transfer of information and results within the LES community. More specifically, this 
evaluation found that dissemination is something done ‘after the fact’ and often 
undertaken in a passive way (i.e., through websites) as opposed to organizations investing 
resources in targeted outreach. Conversely, there is limited awareness as to what projects 
are being funded by ALLESP given that OLES does not publish a list of projects that are 
funded (as was done under the NLP), what their outputs (tools etc.) and outcomes are, the 
quality of the work funded and the extent to which tools can be used more broadly in the 
LES community. The Program does not have a formal mechanism for rolling up the 
lessons learned and best practices developed by its funded projects for the benefit of all in 
the LES community. 

Without effective communication of project outputs and outcomes and monitoring of 
dissemination, the impact of the knowledge, tools and information developed through 
funded projects is minimized. It was noted by a few internal key informants that the 
Program hopes or expects the core funded literacy coalitions and organizations will take 
some responsibility for the dissemination of the outputs from their funded projects as well 
as from their awareness of the activities and outputs being undertaken in their jurisdictions. 
Unfortunately, without direct accountability or funding to perform this activity, funded 
organizations do not seem to be pursuing this in a structured or effective manner. 
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It is recommended that the Program investigate various ways of supporting the transfer 
of the knowledge regarding funded projects and their results. 

More effective and structured mechanisms should be developed and implemented to 
share information about funded projects. These include the identification of projects 
funded, their outputs (i.e. tools, research, etc.) and the outcomes and impacts from 
these outputs, the quality of the work funded (i.e. the degree to which findings can 
be replicated or adapted to different environments) and the extent to which tools 
can be used more broadly in the LES community. In addition, lessons learned could 
be identified to ensure that knowledge, tools and materials are being used to their 
full extent, and to avoid overlap and duplication. These efforts should include 
identifying what information should be disseminated, developing a common approach 
to the collection of the information from funded projects and determining the audiences 
and the method of information dissemination that is appropriate for each group. 

It is suggested that an LES community-based portal be developed where the outcomes 
from the funded projects are shared and community users are able to access the tools 
and research and comment on their relevance and applicability to different settings. 
This approach would be more community driven and allow for more disciplined 
review and dialogue on the quality and usefulness of ALLESP funded projects, 
beyond the self-reporting that will occur as part of the APR and PRR templates. 
The site could also be expanded to include other tools and research developed in the 
broader community. 

Achievement of Immediate outcomes 

This evaluation noted that it is too early in the Program lifecycle to answer the full extent 
to which results have been achieved. This ability was impacted by the fact that there were 
some projects that had not yet ended. 

Nevertheless, this evaluation identified a number of unintended negative impacts that 
have emerged with the creation of ALLESP including: weakened collaboration and 
partnership and communications; ongoing program changes and limited awareness of the 
value of realignment within the LES community; perceived reduction in proposal 
submissions; delayed decision-making and funding approval; administrative burden and a 
reduction of proposal submissions. In many cases, program weaknesses identified in the 
predecessor organizations continue to exist - particularly in relation to database 
weakness, performance monitoring, dissemination of funded project research and tools 
and demonstrating reach or impact. 
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Appendix A - Program Logic Model 
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REACH: (1) families; (2) communities; (3) educational institutions; and (4) workplaces. 











Appendix B - Evaluation Issues 
and Questions Matrix 



Questions 


Indicators 


Methods 47 


1. Program Design and Delivery 


Are ALLESP's objectives and 
expected results clear? Is the 
logic model still valid? 


• Extent to which ALLESP 
activities, outputs and results 
are appropriately identified, 
defined and logically linked 


• Key Informant Interviews 

• Document Review 


Is ALLESP being implemented 
as intended? 


• Extent to which the initiative 
is implemented as intended/ 
differences, etc 


• Key Informant Interviews 

• Document Review 


Has the Program re-alignment 
under ALLESP led to effective 
Program integration, co- 
ordination and coherence? 


• Extent to which Program re- 
alignment led to Program 
integration, co-ordination and 
coherence including but not 
limited to views and 
documented evidence on : 
o the integration of the 
previous Programs under 
ALLESP 

o the clarity of the new 
Program mandate and 
orientation 

o the extent of shared vision 
on the Program by staff 
o the effectiveness of 
Program co-ordination 
o the level of Program 
coherence 


• Key Informant Interviews 

• Document Review 


Who are the recipients of 
ALLESP funding? 


• # and profile of grants and 
contributions applicants 

• # and profile of grants and 
contributions recipients 

• # and profile of core funding 
recipients 

• # and profile of rejected 
applicants 

• Profile of the end-users 
targeted by ALLESP funded 
projects and of outputs. 


• Review of administrative files 
and data 

• Key Informant Interviews 

• Document review 

• Survey of Funding Applicants 



47 All personal information obtained in this evaluation, through key informant interviews and the survey, shall be 
gathered and protected in accordance with the Privacy Act and HRSDC’s Privacy Management Framework, 
including the Governance Protocol for Conducting Policy Analysis, Research and Evaluation Activities. 
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Questions 


Indicators 


Methods 47 


Are adequate management and 
administrative systems in place 
for effective and efficient 
Program delivery? Could 
delivery be improved? 


• Adequacy of management 
and administrative systems in 
place (based on opinions and 
documented evidence) 

• Extent to which the Program 
is delivered efficiently and 
effectively including but not 
limited to issues such as: 

o Satisfaction of applicants 
with application process 
and funding decision 
o Extent to which criteria 
and application process 
for funding are clear and 
well understood 
o Extent to which the 
information on ALLESP is 
available and accessible 
o Views of recipients on the 
management and 
reporting requirements 
(burden, capacity) 
o Turnaround time for 
processing applications 
(communication, decision, 
disbursement, etc.) 

• Views on potential improve- 
ments to delivery process. 


• Review of administrative files 
and data 

• Key informant interviews 

• Document review 

• Survey of funding applicants 


2. Performance Monitoring and Measurement 


Is an adequate performance 
measurement mechanism and 
system in place? How could 
performance measurement be 
improved? 


• Adequacy of mechanism and 
system in place including but 
not limited to issues such as: 

Accuracy: 

o Extent to which the 
performance indicators 
accurately reflect outputs 
and results 

o Extent to which ALLESP 
data capture and reporting 
capacity (including external 
measurement project 
reporting) correspond to 
expectations outlined in 
the performance 
measurement framework 
Quality: 

o Extent to which the 
performance data being 
collected is accurate and 
complete 


• Review of administrative files 
and data 

• Key informant interviews 

• Document review 
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Questions 


Indicators 


Methods 47 




Availability: 

o Extent to which information 
and data can be collected 
Usefulness: 
o Extent to which the 
performance data 
supports decision-making 
and departmental 
accountability 
requirements 

o Perceived usefulness of 
performance data and 
reporting 

How Program performance 
measurement should be 
adjusted (based on opinions 
and analysis of documented 
evidence) 




Is the basic information 
necessary for the summative 
evaluation being collected? 


• Extent to which the data 
related to next evaluation is 
collected and available. 
o Information available and 
collected on applicants 
o Information available and 
collected on recipients 
o Information available on 
projects outputs 
o # of applicants in ALLESP 
administrative data with 
complete contact 
information (mail, 
telephone, e-mail) 
o # of recipients in ALLESP 
administrative data with 
complete contact 
information (mail, 
telephone, e-mail), 
o Etc. 


• Review of administrative files 
and data 

• Key informant interviews 

• Document review 

• Survey of Funding Applicants 
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Questions 


Indicators 


Methods 47 


3. Preliminary Findings - output oriented 


What type of Funding 
Agreements has been entered 
into under ALLESP? 


• # of agreements for (or by): 
o Multi-year funding of LES 
P/T coalitions and 
national organizations 
o Projects that build 
knowledge about 
Canadians’ LES needs 
o Projects that develop 
practical tools and 
supports to respond to 
identified needs 
o stream (workplace/family- 
community) 

o provinces or territories 
o target group 
o etc. 


• Key Informant Interviews 

• Review of administrative files 
and data 

• Survey of Funding Applicants 



64 
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